Contrastive Latent Variable Models for Neural Text Generation
Abstract
Deep latent variable models such as variational autoencoders and energy-based models are widely used for neural text generation.
Most of them focus on matching the prior distribution with the posterior distribution of the latent variable for text reconstruction.
In addition to instance-level reconstruction, this paper aims to integrate contrastive learning in the latent space, forcing the latent variables to learn high-level semantics by exploring inter-instance relationships.
Experiments on various text generation benchmarks show the effectiveness of our proposed method.
We also empirically show that our method can mitigate the posterior collapse issue for latent variable based text generation models.
INTRODUCTION
Deep latent variable models such as variational autoencoder (VAEs)
[Bowman et al., 2015] and deep energy-based models [Deng et al., 2020, Pang et al., 2020] have been widely used for text generation applications, such as machine translation [Calixto et al., 2019], language modeling
[Deng et al., 2020], dialogue understanding [Shen et al., 2017] and story generation [Chen et al., 2021, Jhamtani and BergKirkpatrick, 2020].
These models follow a sequence-tosequence [Sutskever et al., 2014] task setting, ﬁrst stochastically mapping the input sentence into a latent variable according to proper probabilistic distribution assumptions and then reconstruct the whole sentence.
By manipulating the latent variable, controllable text generation models with both diversity and ﬂuency can be achieved.
In such models, the latent variable is expected to encode high-level semantic and style information of the input sequence, which can serve to guide the generation of output sequences in different text generation tasks.
[Table]
1 xx T
LSTM Encoder with dropout
Posterior
Prior
MLP h
LSTM Decoder MLP f h T
MLP
Posterior
Prior
MLP h
MLP
Contrastive Loss
Noise f , ...,, x 1 xx T
LSTM Encoder with dropout , ...,, x 1 xx T , ...,,
LSTM Decoder x 1 xx T , ...,,
In this paper, we consider VAE and its variant iVAE
[Fang et al., 2019], which addresses two limits of simple VAEs.
The ﬁrst is the simple posterior, which is usually assumed to be isotropic Gaussian, limiting the expressive power of the latent variable.
Many efforts have been made to use an enhanced posterior [Casale et al., 2018, Tomczak and
Welling, 2018].
Another problem is the posterior collapse issue [Bowman et al., 2015], where the learned latent variable can be largely ignored in a trained model.
iVAE solves these problems by using implicit representation of posterior distributions of latent variables, where the posterior distribution are represented by a set of samples.
Mutual information maximization is considered to encourage the correlation between the inputs and the induced latent variables.
It has been shown that the strength of iVAE as compared to vanilla VAE lies in better latent variable representations [Fang et al., 2019].
In particular, the latent variable should faithfully represent the semantics of the input, while allowing diversiﬁed outputs.
Intuitively, further improvements towards these two characteristics in the latent variable can potentially lead to stronger VAE methods.
However, there is an intrinsic tradeoff between diversity and faithfulness, as it is more challenging to ensure that a set of different latent representations all embody the same semantic information.
To this end, both VAE and iVAE make use of the reconstruc- tion objective, learning to reconstruct the same input from different latent variable samples.
The effectiveness can be limited by the number of samples that can be drawn, which can hardly match the number of possible variations in the reconstruction output.
To address this issue, we consider adding direct supervision signals on the latent representation itself, by ensuring that different latent variable samples of the same input are closer in the vector space as compared to latent variable samples of two different sentences.
This loss ensures semantic faithfulness without losing diversity.
It can be viewed as belonging to the category of contrastive loss [Khosla et al., 2020, Le-Khac et al., 2020].
Our model structure is shown in Figure 1.
In particular, for
VAE, we encode the same input sentence twice with different dropout masks and then sample the corresponding latent codes for obtaining positive pairs.
Negative pairs are constructed by comparing the sampled latent vector with the remaining latent vectors resided in the same batch.
Contrastive learning is used to increase the semantic similarity of positive pairs and decrease the similarities of negative pairs.
For iVAE, we observe that the KL divergence between the implicit posterior distribution and the prior distribution can explode.
The main reason is that, to calculate the KL divergence between sample-based representations, iVAE uses an approximation based on Fechel duality theorem [Rockafellar et al., 1966, Dai et al., 2018].
However, this approximation cannot guarantee the non-negativity of KL divergence.
We empirically observe that contrastive learning on the latent space can ease the explosion problem.
We further adapt this approximation by borrowing the idea from
CircleLoss
[Sun et al., 2020] to ensure the non-negativity of the KL approximation.
We conduct experiments on a synthetic dataset and three language modeling benchmarks.
Results show that our model can learn better latent representations on the synthetic dataset and achieve better perplexity scores than
VAE and iVAE.
For example, on PennTreebank language modeling benchmark, our model can decrease the perplexity scores by more than 10 points compared to iVAE.
Including contrastive learning over latent variable in VAEs can alleviate the posterior collapse issues and avoid approximated KL divergence explosion in iVAEs.
To our knowledge, we are the ﬁrst to combine contrastive learning with latent variable models for natural language processing.
We make our models and codes publicly available at https://github.com/zeeeyang/ constrastive_vae and an alternative implementation using MindSpore 1 which is a new deep learning computing framework, can be found at https://github.com/ zeeeyang/constrastive_vae_mindspore .
RELATED WORK
Varational Auto-Encoders for Text Generation VAE [Bowman et al., 2015] proposes the ﬁrst model to enable text generation from continuous space using variational inference and an isotropic Gaussian prior.
Many efforts have been made to improve VAE by using advanced prior distributions [Tomczak and Welling, 2018, Wang and Wang, 2019, Ding and Gimpel, 2021] and alleviate the the posterior collapse issue of VAE
[Bowman et al., 2015, Higgins et al., 2017, He et al., 2019, Fu et al., 2019], making the generation depends on latent representations.
iVAE [Fang et al., 2019] uses implicit sample-based representation, without requiring an explicit density form for the approximate posterior, which enables more ﬂexibility.
To solve the posterior collapse issue, iVAE adopts a mutual information regularization to match the aggregated posterior to the prior distribution.
APO-VAE [Dai et al., 2018] deﬁnes both the prior and posterior of latent variables over a Poincaré ball in hyperbolic space, which also adopts the training scheme of iVAE.
We do not compare with Apo-VAE since it additionally adopts a data-dependent VampPrior [Tomczak and Welling, 2018].
Our model is based on iVAE, with direct supervision of latent variables using contrastive learning.
Contrastive Learning for Sentence Representations Contrastive learning
[Oord et al., 2018, Hjelm et al., 2019, He et al., 2020, Chen et al., 2020] has achieved great success in self-supervised visual representation learning.
Recent work transfers this learning strategy to texts with different network architectures and augmenting methods for unsupervised sentence representation learning [Zhang et al., 2020,
Giorgi et al., 2021, Carlsson et al., 2021, Gao et al., 2021].
Among these, SimCSE [Gao et al., 2021], which only uses standard dropout as minimal data augmentation, achieves the state of art and even performs on par with previously supervised counterparts.
Our method uses the same data augmentation method as SimCSE, applying dropout to the input batch twice to obtain two different views.
Similarly, Rdrop [Liang et al., 2021] applies dropout twice to regularize the behaviours of the decoders in language modeling.
However, different from SimCSE and R-drop, the goal of our method is to better supervise the latent state s of variational auto-encoders.
Contrastive Learning for Text Generation Dai et al.
[2021] use CPC to do utterance-level contrastive learning between the dialogue context and the corresponding response.
Su et al.
[2022] propose a contrastive training objective for text generation, which is used to improve MLE based training and beam search decoding.
In their methods, the hidden vector of a token is contrasted with the hidden representation vectors of the remaining tokens in the same sentence, which is different from our method.
To our knowledge, we are the ﬁrst to directly apply contrastive learning over the latent vectors for variational auto-encoders.
Lee et al.
[2021] combines adversarial perturbations with contrastive learning to solve the exposure bias problem for conditional sequence generation.
Their contrastive learning is done in the embedding space, which is different from our method.
METHOD
Figure 2 shows the models we investigated in this paper.
The blue boxes denote our proposed models.
We introduce the VAE baselines in Sec 3.1.
In Sec 3.2, we describe iVAE, which considers sample-based posterior distribution with
VAE, and a variant of iVAE (iVAE MI ) which considers mutual information between input data and latent.
In Sec 3.3, we ﬁrst show two models cl VAE and cl VAE MI by integrating contrastive learning over latent variables for text generation based on iVAE and iVAE MI , respectively.
Then, we discuss cl VAE + and cl VAE + MI , which solve the explosion problem of approximated KL divergence by using circle loss enhancement.
3.1 VARIATIONAL AUTO-ENCODER BASELINE
Formally, given a sentence x = { x 1 , x 2 , . . .
, x T } , where
T is the length of x , an auto-regressive language model ﬁnds a neural network θ that can maximize the loglikelihood log P ( x ; θ )
=
(cid:81) ni
=1 log P ( x i | x 1 , . . .
, x i − 1 ; θ ) .
Variational auto-encoder (VAE) introduces a random latent variable z to model P ( x ; θ )
= (cid:82) z P ( z ; θ ) P ( x | z ; θ ) d z .
To generate a sentence x , a z is ﬁrst sampled from the prior distribution P ( z ; θ ) and then a decoder network is used to produce x from z according to P ( x | z ; θ ) .
In this paper, z is is continuous, log P ( x ; θ ) becomes intractable due to the integration over z .
To train θ , a variational posterior distribution Q ( z | x ; φ ) over z is introduced to approximate the true posterior P ( z | x ; θ ) .
To ensure that z from the prior
P ( z ; θ ) can encode meaningful semantic representations of the input sentence, Q ( z | x ; φ ) is forced to match P ( z ; θ ) during training.
Based on Q ( z | x ; φ ) , an evidence lower bound (ELBO)
[Bowman et al., 2015] to log P ( x ; θ ) is deﬁned as:
[Equation]
[Equation]
θ and φ are jointly trained by maximizing ELBO in Eq 1, where the ﬁrst term is a reconstruction loss.
The encoder ﬁrst generates z according to Q ( z | x ; φ ) , which is called an encoder network or a recognition network .
z is required to reconstruct x by maximizing log P ( x | z ; θ ) .
The second term of Eq 1 is the Kullback-Leibler divergence, which regularizes the posterior distribution to be close to the prior distribution.
The posterior distribution and the prior distribution are often assumed to belong to the same parametric distributions with different parameters, which can ease the learning burden.
Ideally, a good model maximizes the ﬁrst term and minimizes the second term.
When the KL term tends to be 0, Q ( z | x ; φ ) degenerates to P ( z ; θ ) and the posterior collapse issue happens.
For text generation, it is common due to the auto-regressive decoder can be very strong, which totally ignores z and generates x i solely from [ x 1 , . . .
, x i − 1 ]
[Fu et al., 2019].
3.2 IMPLICIT VAE BASELINE (IVAE)
Q ( z | x ; φ ) is typically assumed to be multivariate Gaussian distributions for computational convenience.
However,
Gaussians are not enough to capture the rich semantics of natural sentences in latent space.
Implicit VAEs (iVAE) introduce a sample-based posterior representation which does not depend on an explicit density form.
Q ( z | x ; φ ) is represented by a set of samples { z x ,i } Si =1 , where S is the sample size.
The i -th sample is given by
[Equation]
where N ( ξ ) is the standard Gaussian, E NC ( x , ξ i ; φ ) is a noise aware encoder network.
The sentence representation h T of x is generated by a LSTM encoder [Hochreiter and
Schmidhuber, 1997]and h T is concatenated with ξ i by a multi-layer perceptron layer (MLP) to produce z x ,i .
The KL divergence of Eq 1 is intractable after using the sample based posterior representation.
Following Fang et al. [2019], its dual form is calculated according to Fenchel duality theorem [Rockafellar et al., 1966, Dai et al., 2018] by introducing an auxiliary function f ( x , z ; ψ ) ,
[Equation]
where f outputs a real value and ψ is the model parameters for f .
f is implemented as a MLP, which distinguishes between ( x , z Q ) and ( x , z P ) where z Q and z P denote latent samples drawn from the posterior and the prior distributions, respectively.
Using Eq 3, ELBO can be written as
[Equation]
Considering a dataset D
= { x i } ni =1 , the loss function of the whole D using VAE and iVAE are:
[Equation]
Mutual Information Regularized iVAE (iVAE MI )
In
Eq 1, the KL term forces a data-dependent posterior
Q ( z | x , φ ) to match the same data-agnostic prior distribution P ( z ; θ ) .
An variant of iVAE uses aggregated posterior Q ( z ; φ ) to match P ( x | z ; θ ) , where Q ( z ; φ ) = (cid:82)
Q ( x ) Q ( z | x ; φ ) d x and Q ( x ) is the empirical data distribution.
Using aggregated posterior Q ( z ; φ ) , the latent space can be better regularized by cooperating different posterior distributions of all sentences to jointly match the prior.
Given a dataset D = { x i } ni =1 , replacing the expected KL term E x ∼ D KL ( Q ( z | x ; φ ) , P ( z ; θ )) with
KL ( Q ( z ; φ ) , P ( z ; θ )) , the new training objective is,
[Equation]
where g is a similar function as f to produce a real value.
The dual form of KL ( Q ( z ; φ ) , P ( z ; θ )) is max g E z ∼ Q ( z ; φ ) g ( z , ψ ) − E z ∼ P ( z ; θ ) exp ( g ( z , ψ )) .
Differently, g only considers z instead of the concatenation of x and z .
To generate samples from Q ( z , φ ) , ancestral sampling can be used.
First, x is sampled from D and then z is sampled from Q ( z | x ; φ ) .
Fang et al. [2019] show that optimizing the aggregated posterior based KL term is to maximize the mutual information
I ( x , z ) under the joint distribution Q ( x , z ; φ ) .
We take the above model ( iVAE MI ) as our main baseline.
3.3 CONTRASTIVE LEARNING OVER LATENT VARIABLES FOR TEXT GENERATION
As shown in Figure 1, the overall structure for our proposed model is based on the framework of iVAE.
It contains three main components, an encoder, a contrastive learning module and a decoder.
The encoder is quite similar to iVAE, except that dropout is enabled.
The decoder is exactly the same as iVAE.
The contrastive learning module is our main design.
In this section, we show how to integrate contrastive learning over latent variables with iVAE.
Formally, given the latent sample z i of the i -th input sentence, its positive latent sample z p = { z + i } which is semantically similar to z i , its M negative latent samples z n = { z − i,j } Mj =1 which is semantically far away from z i , the training objective using contrastive learning for the i -th sentence is to maximize:
[Equation]
where τ is a temperature hyper-parameter and S IM ( z 1 , z 2 ) is a function for measuring semantic similarities between zand z 2 .
Particularly, we use cosine similarity as the semantic similarity metric,
[Equation]
where (cid:107) z (cid:107) 2 denotes the L 2 norm of z .
Following Chen et al.
[2020] and Gao et al.
[2021], we consider batch-based contrastive learning.
For each sentence x i , we follow Gao et al.
[2021] and use dropout as data augmentation.
We encode the input sentence twice with dropout being enabled, and obtain two different views x i and x + i of the same sentence.
x i is then regarded as the anchor sentence and x + i is the positive sample of x i .
The remaining sentences in the same batch are treated as the negative samples of x i .
In this way, the number of negative examples M = N − 1 .
To obtain the corresponding posterior latent variables, we sample z i ∼ Q ( z | x i ; φ ) and z + i ∼ Q ( z | x + i ; φ ) .
For the posterior samples z j ∼ Q ( z | x j ; φ ) and j (cid:54) =
i , they are the negative samples of z i .
Denote u i = {
z + i } ∪ { z − i,j } Mj =1 and the overall contrastive learning loss for the whole batch is
[Equation]
Adding the contrastive loss into L I VAE and L I VAE MI , we have
[Equation]
Improved Dual Function KL ( Q ( z | x ; φ ) , P ( z ; θ )) should be non-negative according to its deﬁnition.
However, the approximation by the dual function in Eq 3 cannot ensure this property since E z ∼ Q ( z | x ; φ ) f ( x , z ; ψ ) can be less than E z ∼ P ( z ; θ ) exp ( f ( x , z ; ψ )) .
To encourage the KL approximation to be non-negative, we resort to a new constraint E
[Equation]
Using this constraint [Equation] we deﬁne a new approximation inspired by circle loss [Sun et al.
[Equation] [Equation]
To minimize
[Equation]
should be forced to be much greater than 0 and E z ∼ P ( z ; θ ) f ( x , z ; ψ ) should be much less than 0.
Similarly, KL ( Q ( z ; φ ) , P ( z ; θ )) can be approximated as,
[Equation]
Using the proposed KL approximation term
[Equation]
the full objective is to maximize
[Equation]
Training Algorithm Algorithm 1 shows the training algorithm of our proposed method.
We ﬁrst sample a mini-batch of paired random Gaussian noise vectors ξ i and ξ + i .
After obtaining a mini-batch of input sentences, we pass them through the LSTM encoder with dropout being enabled to produce the latent vectors z x ,i and z + x ,i .
Then the contrastive loss deﬁned in Eq 10 is calculated, and a paired prior vectors are sampled from P ( z ; θ ) .
ψ in f ( x , z , ψ ) is updated according to Eq 18.
Here we further consider to minimize the differences between the dual functions of the same input data by deﬁning a squared loss L SQ , which is given by
[Equation]
Given the same input and different z , the values of function f are forced to reside in a small region.
ψ is ﬁxed afterwards.
The encoder parameters φ and decoder parameters θ are updated according to Eq 19.
If the dual function g instead of f is used, a similar loss as Eq 17 can be deﬁned using g , and Eq 18 and Eq 19 can be changed accordingly.
EXPERIMENTS
We evaluate our proposed models on two tasks: 1) language modeling; 2) a synthetic clustering setting;
4.1 DATA AND SETTINGS
Language Modeling For language modeling, we consider three datasets, including the Penn Treebank [Marcus et al., ξ i ∼ N ξ )
ξ + i ∼
mini
batch sentences x Generate z x ,i = E NC ( x i , ξ i ; φ ) and z + x ,i =
E NC ( x i , ξ + i ; φ ) ; 4.
Calculate L CL in Eq 10; 5.
Sample a mini-batch of z i , z
+ i ∼ P ( z ; θ ) ; 6.
Update ψ in
f
[Equation]
) to minimize [Equation] 7. Update parameters φ , θ
to minimize
[Equation]
[Equation]
Output : θ , φ and ψ
[Equation]
1993], Yahoo [Yang et al., 2017] and Yelp
[He et al., 2019].
The Penn Treebank (PTB) is a common benchmark for language modeling, which consists of 42K sentences of varying lengths and 1 million words of Wall Street Journal material in 1989.
Yahoo and Yelp are much larger datasets compared to the PTB.
They both consist of 100K sentences and the average length of them are 78.7 and 96.0 words, respectively.
Each dataset contains train, validation and test sets.
The detailed data statistics are in Appendix A.1.
We select the best model according to the performance on validation set and report the results on the test set using the corresponding best model.
Synthetic Clustering Data Following Fang et al. [2019], we design an experiment of synthetic clustering to show the learning dynamics of the induced latent variable.
Given a random input category k , k ∈ [0 , K − 1] and K is the number of categories, a 2-dimensional random Gaussian noise sampled from N (0 , 1) is combined with the one-hot representation of the input category as the input variable x .
An encoder with multi-layered feedforward neural networks transforms x into a 2-dimensional latent vector z .
An decoder is trained to reconstruct x from z .
Suppose that the reconstructed output is y and y ∈ R K , the binary cross-entropy loss is used, namely
[Table]
L = − e k log σ ( y k )
+ (cid:80) i (cid:54) =
k (1 − e i ) log(1 − σ ( y i )) , where k is the input category, e is the corresponding one-hot vector representation of the category k
and σ is the sigmoid function σ ( x ) = 1 1+exp( − x ) .
During training, we generate a batch of one-hot vectors by randomly choosing from [1 , 0 , 0 , 0] , [0 , 1 , 0 , 0] , [0 , 0 , 1 , 0] and [0 , 0 , 0 , 1] when K = 4 .
The batch size is
and we train the encoder-decoder 30K to 80K epochs until convergence.
Ideally, the cluster of latent variable z in 2 dimensional space should exactly correspond to the input category after training.
To understand the training behaviour of our proposed model, we visualize z in 2D space.
4.2 LANGUAGE MODELING RESULTS
PTB Results We compare our models with state-of-the-art
VAE language models without using contrastive learning, including: 1) traditional VAE models with a KL-annealing strategy [Bowman et al., 2015]; 2) β VAE [Higgins et al., 2017], which controls the penalty on KL using a small hyper-parameter β ; 3)
Sa VAE
[Kim et al., 2018], which is a semi-amortized VAE; 4) Cyc VAE
[Fu et al., 2019], which anneals the KL term in a cyclical way; 5) lag VAE [He et al., 2019], which lags the update of decoder by aggressively updating encoder several times; 6)
i VAE [Fang et al., 2019], which assumes an implicit distribution of latent variable as mentioned before; 7) i VAE MI is an enhanced version of i VAE by directly considering the mutual information between the input x and the latent variable z .
We denote our model as cl VAE
(Eq 11) and the circle loss enhanced version is named as cl VAE
+ (Eq 15).
cl VAE MI (Eq 12) and cl VAE + MI (Eq 16) are their corresponding enhanced versions by mutual information.
We evaluate the models in terms of the quality of both the generation texts and the induced latent features.
To measure the generation outputs, we use negative ELBO scores (-
ELBO) and perplexity scores (PPL).
The lower they are, the better the generated outputs are.
Following Fang et al. [2019], we evaluate the quality of induced latent features in terms of KL ( Q ( z | x ; φ ) || P ( z ; θ )) , the mutual information
[Table]
I ( x , z ) under the joint distribution of Q ( x , z ; φ ) , and the number of active units of z , which is deﬁned as AU ( z ) =
C OV x ( E z ∼ Q ( z | x ; φ ) [ z ]) > 0 .
Since all these terms cannot be solved in an analytical way, they are approximated by sample z 128 times from Q ( z | x ; φ ) .
In general, higher KL, larger MI and AU values indicate the latent feature is more sufﬁciently used by the model.
Table 1 shows the main results on PTB test set.
Our simplest model cl VAE outperforms all the baselines, which shows that adding contrastive learning over latent variables greatly improves the model performance.
cl VAE reduces
ELBO and PPL by 4.9 and 10.95 points compared to i VAE.
The KL and MI values are increased by 2.05 points compared to i VAE.
There results show that our model make better use of the latent vector compared to baselines, and produce better outputs.
Similar conclusions hold for the models enhanced by mutual information, which suggests that contrastive learning can be complementary to mutual information estimation.
Particularly, cl VAE MI gives the best
KL and MI values among all the models, demonstrating that it is a good choice to use the learned latent features.
Using the proposed circle loss enhancement, the model performance improves with respect to ELBO and PPL, with 79.
ELBO and 38.04 PPL, which are much better than cl VAE (82.7 ELBO and 43.51 PPL).
Among them, cl VAE + MI gives the best results, presenting 77.7 ELBO and 34.61 PPL.
In terms of KL and MI, using the circle loss enhancement slightly hurts the sufﬁcient use of latent variables compared to cl VAE MI .
However, the KL and MI values are relative large, ranking the third among all the models.
Therefore, considering both the generation quality and the effective use of latent space, the circle loss enhancement is useful.
Yahoo and Yelp
Results We use the best conﬁguration model of cl VAE + MI to conduct experiments on Yahoo and
Yelp.
Table 2 shows the results.
On Yahoo, our model outper-
[Table] [Table]
form i VAE MI in terms of both ELBO and PPL.
For ELBO, our model outperform i VAE MI by 5.8 points.
For PPL, our model gives 44.61 points, reducing the PPL of i VAE MI by 3.32 points.
For KL and MI values, our model are better than i VAE and comparable to i VAE MI .
Similar observations can be found on Yelp dataset.
cl VAE + MI reduces ELBO and
PPL by 5.1 and 1.91 points compared to i VAE MI .
On Yelp, for KL and MI values are 8.8 and 8.2, respectively, ranking the second.
Since both Yahoo and Yelp are large corpora, it is challenging to greatly reduce the PPL and ELBO scores while maintaining high KL and MI values at the same time.
Our model is empirically well balanced.
4.3 ANALYSIS
The effect of contrastive learning for latent variable Table 3 shows the comparison results of doing contrastive learning using the encoder output h and the latent variable z on PTB test set, respectively.
For simplicity, we use two basic models: namely cl VAE and cl VAE MI .
As shown in Table 1, when the same models are used, doing contrastive learning over z instead of h gives better results.
For example, when using the cl VAE MI model, the model based on h gives 52.
PPL, while the model based on z obtains 43.49 PPL, which is much lower than the model based on h .
Only in terms of ELBO and PPL values, the basic model cl VAE over z even performs better than the mutual information enhanced cl VAE MI models over h .
Besides, when doing contrastive learning on h , the performance gap between cl VAE and cl VAE MI are large, whereas the corresponding gap is shortened when doing contrastive learning on z .
These results suggest that it is necessary to do contrastive learning over latent variables, producing better results compared to directly doing contrastive learning on the encoder hidden output.
The effect on the decoder To evaluate whether the decoder
[Figure]
is being improved by making better use of latent features, we sample latent variables from the prior distribution P ( z ) and ask the decoder to generate outputs based on the sampled latent codes, following Fang et al. [2019] and Kim et al.
The generated text is evaluated by KenLM [Heaﬁeld et al., 2013] using two metrics: forward PPL and reverse PPL.
The forward PPL evaluates the generated texts based on a language model trained on the PTB-train corpus.
Lower forward PPL indicates the generated texts are more ﬂuent.
The reverse PPL evaluates the PTB corpus based on a language model trained on the generated texts.
Lower reverse PPL means the generated texts are better representative of the PTB corpus.
For the underlying language model, 5-gram KenLM is used.
Table 4 shows the forward PPL and reverse PPL on the PTB.
For reverse PPL, our cl VAE + performs better than all the baselines and cl VAE MI gives the best values, which shows that our models can better represent the true data distribution.
For forward PPL, our model performs better than most of baselines models including Sa VAE and β 0 .
5 VAE.
iVAE gives the best forward PPL scores (116) since it imposes less constraints on the latent variable and thus beneﬁts from a smoother space.
Our model give 120 forward PPL scores, which is comparable to iVAE and acceptable by sacriﬁcing a little ﬂuency, but it can generate more diverse outputs and make text appear more human like.
The negativity of approximate KL and MI terms Section 3.3 mentioned that the approximated KL and MI term can become negative during training iVAE.
Empirically, we demonstrate this problem by using the PTB language modeling task.
Figure 3a and Figure 3b show how the KL term and the mutual information term change with the training epoch of the three models, including iVAE, cl VAE and cl VAE + MI , respectively.
As shown in Figure 3a, the approximate KL term of iVAE suddenly becomes a large negative number
[Table]
at the 39th epoch.
From Figure 3b, we can observe that the approximate MI term at 0th epoch, 2nd epoch and the 39th epoch are all negative.
This observation shows that the negativity problem of approximate KL and MI terms do exist in iVAE.
Adding the contrastive learning module alleviates this problem to some extent.
The KL and MI values of cl VAE become more stable than iVAE.
However, the KL value of cl VAE dramatically decreases at the 10th epoch.
In contrast, no such phenomenons exist in our cl VAE + MI model based on the empirical observations.
Its KL and MI values gradually increase as the training goes on in general.
These evidences suggest that our proposed circle loss enhancement can learn a more robust model in term of the stability of the approximated KL and MI values.
More Analysis More analysis on sentence interpolation and effect of different dropout rates are in A.2 and A.3.
4.4 RESULTS ON SYNTHETIC DATA
K =
Table 5a shows the visualizations of the learned 2D latent vector of VAE, iVAE and our model with different training epochs.
When epoch =
K , the distribution of the induced latent variable of VAE is basically a normal distribution and the points are mixed together.
Since both iVAE and our model represents the latent variable using an implicit distribution, there are no clues about the distribution of the latent variable in the beginning.
After training with 5 K epochs, we observe that VAE can make an initial guess concerning the corresponding cluster of the induced latent variable, iVAE is still struggling to cluster these points, whereas our model can achieve a clear separation between clusters, and capture well the latent distribution of the data.
As the training continues to 15K and 30K epochs,
VAE shapes the data distribution as a sandwich, as shown in Table2, the green and blue points are widely mixed together and the data points within the same cluster are wide spread.
Meanwhile, iVAE still could not provide a nearly clear separation between clusters.
However, our model starts to shorten the distance among points within the same cluster and enlarge the distance between different clusters.
As a fact, our model only needs 3.5K epochs to converge while both VAE and iVAE take 80K epochs to converge.
After 80k epochs, iVAE divides the 2D space into the corresponding clusters successfully, and it can provide a better data distribution compared to VAE.
However, the data points with the same cluster given by iVAE are dispersed.
As shown in Table 5a, our model converges much faster than VAE and iVAE.
It also gives a more compact and coherent representation, making the intra-class distance smaller and the inter-class distance larger compared to VAE and iVAE.
This shows that adding contrastive learning over latent spaces can better regularize the distributions of latent variables.
K > 4 When the number of category becomes large, the difﬁculty of representing data points in the 2D latent variable also increases.
Table 5b shows the comparisons of converging trend between iVAE and our model when the input category is larger than K = 4 .
When K = 8 , iVAE fails to converge even using 80 K training epochs, while our model obtains a decent separation boundaries using only 5K epochs.
After training 80 K epochs, our model successfully cluster the input data into the corresponding category.
To show the capability of our method, we further set K = 16 , making it more challenging.
In this setting, the class boundaries appear after 5K epochs, but not so clear as those when
K = 8 using the same training budget.
Separation between clusters gets reﬁned as the iterations go on.
With 80K training epochs, our models can manage to reconstruct the input categories from the 2D latent variable.
This not only shows the discriminatory nature of the learnt representations, but also the speed with which becomes a potential advantages of combining contrastive learning with latent variables, and the capacity of handling more complex situations compared to the traditional latent variable models.
4.5 SENTENCE INTERPOLATION
Table 6 shows the sentence interpolation [Bowman et al., 2015] results of two example sentences.
Similar to VAE, our model can generate sentences by interpolating the latent semantic vectors.
Given two sentences x 1 and x 2 , we generate vectors z 1 and z 2 by averaging samples from
[Table]
Q φ ( z | x ) .
Given z 1 and z 2 , we generate a new latent vector z
= λ ∗ z
+ (1 − λ ) ∗ z 2 by interpolating the sentence semantics of z 1 and z 2 .
Then z is used by the decoder to produce a sentence with mixed semantics.
λ is varied fromto 1 with a step size of 0.1.
As shown in Table 6, our model can generate sentences by smooth considering the semantics from the two input sentences.
4.6 EFFECT OF DIFFERENT DROPOUT RATES
[Figure]
We use the PTB to study the effect of different dropout rates by varying the dropout ratio from 0.1 to 0.9 with a step size 0.1.
Figure 4 shows the PPL and ELBO values of different dropout rates.
We ﬁnd that droput = 0 .
3 gives the optimal performance.
Empirically, increasing the dropout rates from 0.1 to 0.3, the performances become better.
When the dropout ratio is in range [0.3, 0.6], the performances are stable.
Using a large dropout rate ( dropout > = 0 . 7 ), the performances start to decrease.
CONCLUSION
We proposed contrastive learning over latent variables in variational autoencoders (VAE), designing a circle loss enhancement to solve the approximation problem of the KL term used in a state-of-the-art VAE models.
Experiments showed that combining contrastive learning with latent variable models can both improve the generation quality and make sufﬁcient use of latent space.
In future, we intend to expand our framework to graph and image data.
Author Contributions
Zhiyang Teng proposed the main idea, wrote the code and the paper, and performed main experiments.
Chenhua Chen performed auxliary experiments and polished the paper.
Yan
Zhang participated in the discussion of the method of contrastive learning and wrote part of the related work.
Yue
Zhang provided insights to claim the main idea and proofread the writings.
Acknowledgements
Yue Zhang is the corresponding author.
Zhiyang Teng is sponsored by CCAI-Huawei MindSpore Open Fund.
Yue
Zhang is funded by the Zhejiang Province Key Project 2022SDXHDX0003.
We thank all the reviewers for their valuable comments and suggestions.
We gratefully acknowledge the support of MindSpore, CANN(Compute Architecture for Neural Networks) and Ascend AI Processor used for this research.