H IERARCHICALLY -S TRUCTURED V ARIATIONAL A UTOENCODERS FOR L ONG T EXT G ENERATION
Anonymous authors
Paper under double-blind review
A BSTRACT
Variational autoencoders (VAEs) have received much attention recently as an endto-end architecture for text generation.
Existing methods primarily focus on synthesizing relatively short sentences (with less than twenty words).
In this paper, we propose a novel framework, hierarchically-structured variational autoencoders ( hier -VAE), for generating long and coherent units of text.
To enhance the model’s plan-ahead ability, intermediate sentence representations are introduced into the generative networks to guide the word-level predictions.
To alleviate the typical optimization challenges associated with textual VAEs, we further employ a hierarchy of stochastic layers between the encoder and decoder networks.
Extensive experiments are conducted to evaluate the proposed method, where hier -VAE is shown to make effective use of the latent codes and achieve lower perplexity relative to language models.
Moreover, the generated samples from hier -VAE also exhibit superior quality according to both automatic and human evaluations.
I NTRODUCTION
Probabilistic generative models for text have received considerable attention recently, in part because of their ability to leverage abundant unlabeled data and learn text representations with strong generalization abilities.
Combined with the ﬂexibility of deep neural networks, generative modeling provides a powerful and effective framework to estimate the underlying probability distribution of sentences or documents , and to potentially capture the rich sequential information inherent in natural language.
In this work, we focus on one speciﬁc type of generative model: the variational autoencoder (VAE) (Kingma & Welling, 2013).
i went there a few days ago and was told it was hor- rible , i was told there was a horrible customer service , and they were rude, i would not recommend this location to anyone , i would not recommend this location to anyone .
The VAE maps a text sequence into a continuous latent variable, or code , via an inference (encoder) network; a generative (decoder) network is utilized to reconstruct the input sentence, conditioned on samples from the latent code (via its posterior distribution).
The inference and generative networks, parametrized by deep neural networks, can be optimized jointly by maximizing a variational lower bound of the marginal distribution of the training corpus.
The advantages of leveraging the neural variational inference (NVI) framework have been demonstrated on a wide range of natural language processing (NLP) tasks (Bowman et al., 2016;
Miao et al., 2016;
Yang et al., 2017;
Serban et al., 2017; Shah & Barber, 2018; Kim et al., 2018; Kaiser et al., 2018; Bahuleyan et al., 2018;
Deng et al., 2018).
Although NVI-based methods have demonstrated success in modeling (generating) relatively short sentences (typically less than twenty words), there are several challenges that need to be addressed to facilitate long text generation with deep latent-variable models.
The ﬁrst challenge originates from the nature of recurrent neural networks (RNNs), which have become the cornerstone for text generation.
RNNs are typically trained with gradient descent, using backpropagation through time (BPTT)
(Rumelhart et al., 1986).
However, it is very difﬁcult to scale such a training paradigm to long sequences, due to vanishing or exploding gradients (Hochreiter & Schmidhuber, 1997).
Moreover, the information of the entire generated sequence needs to be stored in the RNN intermediate hidden states, which could be very demanding for RNNs with the increasing length of generated texts.
As shown in Table 1, the sentence sampled from a vanilla textVAE model (Bowman et al., 2016) tends to repeat itself as the sequence proceeds, leading to inferior generation results.
In addition to the issue of repetitiveness, to generate globally-coherent long text sequences, it is desirable that both the higher-level abstract features and lower-level concrete details ( e.g. , speciﬁc word choices) of longer texts can be leveraged by the generative network.
This intuition is also hard to capture with a single-layer (ﬂat) RNN-based generative network.
Another underlying challenge of generating long text relates to the “ posterior collapse ” issue inherent in training variational autoencoders with a strong autoregressive decoder (Bowman et al., 2016;
Yang et al., 2017; Semeniuta et al., 2017; Shen et al., 2018).
Bowman et al.
(2016) found that while employing an LSTM-based VAE for text, the posterior distribution of latent codes tends to match the prior distribution regardless of the input sequence (the KL divergence between the two distributions is very close to zero).
Consequently, the information from the latent variable is not leveraged by the generative network (Bowman et al., 2016).
To mitigate this issue, several strategies have been proposed (see optimization challenges in Section 2) to make the decoder less autoregressive; thus, less contextual information is utilized by the decoder network (Yang et al., 2017; Shen et al., 2018).
However, reducing the autoregressive power of the decoder to make the latent variable more informative can be suboptimal from a text-generation perspective, since the coherence of the generated paragraphs may be sacriﬁced.
Motivated by these observations, we propose a hierarchically-structured variational autoencoder ( hier -VAE), a novel variational approach for generating long sequences of text.
To improve the model’s plan-ahead capability for capturing long-term dependency, our approach exploits a hierarchical LSTM decoder as the generative network.
Speciﬁcally, a set of intermediate sentence-level representations are introduced during the decoding process (via a sentence-level LSTM network), providing additional semantic information while making word-level predictions.
To ameliorate the posterior collapse issue associated with training a VAE for text, we further propose leveraging a hierarchy of latent variables between the convolutional inference networks and recurrent generative networks.
With multiple stochastic layers, where the prior of each latent variable is inferred from data, rather than ﬁxed (as in a standard VAE setting (Kingma & Welling, 2013)), the sampled latent codes at the bottom level are endowed with more ﬂexibility to abstract meaningful features from the input sequences.
We evaluate the proposed hier -VAE comprehensively on language modeling, generic (unconditional) text generation, and conditional generation tasks.
The proposed model demonstrates substantial improvement relative to several baseline methods, in terms of perplexity on language modeling and quality of generated samples (measured by both the BLEU statistics and human evaluation).
We further generalize our method to the conditional generation scenario to demonstrate the versatility of the proposed method.
R ELATED W ORK
VAE for text generation
The variational autoencoder, trained under the neural variational inference (NVI) framework, has been widely used for generating text sequences (Bowman et al., 2016;
Yang et al., 2017; Semeniuta et al., 2017; Zhao et al., 2017).
By encouraging the latent feature space to match a prior distribution within an encoder-decoder architecture, the learned latent variable could potentially encode high-level semantic features and serve as a global representation during the decoding process (Bowman et al., 2016).
The generated results are also endowed with better diversity due to the sampling procedure of the latent codes (Zhao et al., 2017).
However, existing works have mostly focused on generating one sentence (or multiple sentences with at most twenty words in total).
The task of generating relatively longer units of text has been less explored.
Our proposed architecture seeks to extend the NVI framework for long-form text generation by exploiting the hierarchical nature of a paragraph (multi-sentence text sequences), where the outputs from a sentence-level LSTM are utilized to guide word-level predictions.
Optimization challenges with text-VAEs The “posterior collapsing” issue associated with training text-VAEs was ﬁrst outlined by (Bowman et al., 2016), where they have utilized KL divergence annealing and word dropout strategies to mitigate the optimization challenge.
However, their resulting method still does not outperform a pure language model in terms of the ﬁnal perplexity.
Yang et al.
(2017) argue that the small KL term relates to the strong autoregressive nature of an LSTM generative network, and they proposed to utilize a dilated CNN as a decoder to improve the informativeness of the latent variable.
(Zhao et al., 2018b) proposed to augment the VAE training objective with an additional mutual information term.
Unfortunately, such a strategy results in an intractable integral in the case where the latent variables are continuous.
Our work tries to tackle the “posterior collapse” issue from two perspectives: 1) more ﬂexible priors are assumed over the latent variables (learned from the data); 2) the hierarchical structure within a paragraph is taken into account, so that the latent variables can focus less on the local information ( e.g. , word-level coherence) but more on the global features.
Hierarchical structures in NLP Natural language is inherently organized in a hierarchical manner (characters form a word, words form a sentence, sentences form a paragraph, paragraphs from a document, etc .).
Yang et al.
(2016) employed hierarchical LSTM encoders at the word- and sentence-level along with an attention mechanism to learn document representations.
Li et al.
(2015) proposed a hierarchical autoencoder to reconstruct long paragraph of texts.
One method conceptually similar to our work is that of (Serban et al., 2017), which produces a stochastic latent variable for each sentence during decoding.
In contrast, our model encodes the entire paragraph into one single latent variable.
As a result, the latent variable learned in our model relates more to the global semantic information of a paragraph, whereas those in (Serban et al., 2017) mainly contain the local information of a speciﬁc sentence.
Therefore, their model is not suitable for tasks such as latent space interpolation.
Modeling hierarchical structures has been shown to be beneﬁcial for generating others types of data as well, such as images (Sønderby et al., 2016; Gulrajani et al., 2016), music (Roberts et al., 2018), or speech Hsu et al.
H IERARCHICALLY - STRUCTURED VAE M ODEL
Our goal is to build a model that can generate human-generated-like sentences conditioned on random samples from the (prior) latent space.
The proposed method, as illustrated in Figure 1, is built upon the VAE framework introduced in (Bowman et al., 2016), and consists of a hierarchical CNN inference network (encoder) and a hierarchical LSTM generative network (decoder).
The encoder attempts to extract features at different semantic levels, which are then leveraged by the decoder to allow long-term planning while synthesizing long-form text.
We deﬁne long-form text as sequences of sentences (such as a Yelp review on a business, or an abstract of an article) that are arranged in a logical manner and usually follow a deﬁnite plan for development.
3. V ARIATIONAL A UTOENCODER
Recurrent neural network (RNN) language models (Mikolov et al., 2011), which predict each token conditioned on the entire history of previously generated tokens, have been widely employed for unsupervised generative modeling of natural language.
Although effective, they typically do not explicitly capture the global ( high-level ) features ( e.g. , topic or style properties) of a paragraph or sentence.
Motivated by this, the variational autoencoder (VAE) was introduced by (Bowman et al., 2016) as an alternative for generative text modeling, where a stochastic latent variable is augmented as the additional information to modulate the sequential generation process.
Let x denote a text sequence, which consists of L tokens, i.e. , x 1 , x 2 , ..., x L .
A continuous latent code z is ﬁrst sampled from the prior distribution p ( z ) (a multivariate Gaussian prior is typically chosen), and then the text sequence x is generated, conditioned on z , via a generative (decoder) network.
Therefore, the generating distribution can be written as:
[Equation]
where θ are the parameters of the generative network (to be learned).
Since the information from latent variable z is incorporated during the entire generation process, the high-level properties of the corresponding paragraph or sentence could be better leveraged (Bowman et al., 2016;
Yang et al., 2017).
To estimate the parameters θ , one would ideally maximize the marginal distribution p ( x )
= (cid:82) p ( z ) p ( x | z ) d z .
However, computing this marginal is intractable in most cases of practical interest.
Instead, we maximize a variational lower bound, as typically employed in the VAE framework (Kingma & Welling, 2013): L vae =
E q φ ( z | x ) [log p θ ( x | z )] − D KL ( q φ ( z | x ) || p
( z )) .
Note that q φ ( z | x ) is employed to approximate the true posterior, which is usually called the inference or encoder network (parameterized by φ ).
The Kullback-Leibler (KL) divergence term
D KL ( q φ ( z | x ) || p ( z )) , which can be written in closed-form (Kingma & Welling, 2013), encourages the approximate posterior distribution q φ ( z | x ) to be close to the prior p ( z ) .
Note that the VAE prior is often taken to be a diagonal-covariance Gaussian for convenience.
Although the NVI framework has been shown effective for various NLP tasks, directly applying it to long text generation is still challenging.
The reasons are two-fold: (1) the lack of a long-term planning mechanism, which
[Figure]
is critical for generating semantically-coherent long texts (Serdyuk et al., 2017); (2) the posterior collapse issue.
Concerning (2), it was demonstrated in (Bowman et al., 2016) that due to the autoregressive nature of the RNN, the decoder tends to ignore the information from z entirely, resulting in an extremely small KL term.
3. H IERARCHICAL LSTM- BASED GENERATIVE NETWORK
To improve the plan-ahead ability of the generative network, we introduce intermediate sentence representations to facilitate the sequential generation process.
Instead of directly making word-level predictions merely conditioned on the semantic information from z , a series of plan vectors are ﬁrst generated based upon z with a sentence-level LSTM decoder.
The hypothesis underlying this design is that explicitly incorporating the knowledge of (inherently hierarchical) paragraph structure is beneﬁcial for the model to capture sentence-level coherence and potentially mitigate repetitiveness.
Intuitively, while predicting each token, the information from both previously generated words and sentence-level representations can be utilized (see Figure 1).
Assume that the input paragraph consists of M sentences, and the t -th sentence contains N t words (for t = 1 , 2 , ..., M ).
To generate the plan vectors, the sampled latent code z is ﬁrst sent through a one-layer multi-layered perceptron (MLP), with ReLU activation functions, to obtain the starting state of the sentence-level LSTM decoder.
Subsequently, the representations for each sentence, the plan vectors are inferred in a sequential manner: h st = LSTM sent ( h st − 1 , z ) , for t = 1 , 2 , 3 , ..., M .
Note that z is fed to the higher-level LSTM at every time step to predict the sentence representations.
This is because the latent code can be regarded as a paragraph-level abstraction, and thus is informative for determining the semantics of each generated subsequence.
The generated sentence-level plan vectors are then passed through a word-level LSTM decoder to generate the words for each sentence.
Similar to the sentence-level decoder, the corresponding plan vector , h st , is concatenated with the word embedding of the previous word and fed to the LSTM at every time step (see Figure 1).
Let w t,i denote the i -th token in the t -th sentence (teacher forcing is employed during training, and we use greedy decoding at test time).
This process can be expressed as (for t = 1 , 2 , ..., M and i = 1 , 2 , 3 , ..., N i ):
[Equation]
h wt, 0 is inferred from the corresponding plan vector via an MLP layer.
Weight matrix V is used for computing a distribution over words, and W e are word embeddings to be learned.
For each sentence, once the special END token is generated, the word-level LSTM stops decoding (note that each sentence is padded with an END token at the preprocessing step).
The parameters of the word-level decoder for each sentence are shared, and are trained jointly with the parameters of sentence-level LSTM.
Inspired by Chen et al.
, we hypothesize that with the hierarchical decoder networks described above, the latent codes are encouraged to abstract more global and high-level semantic information from the paragraph.
While relatively more detailed information ( e.g. , word-level (local) coherence) can be captured via the word- and sentence-level LSTM networks, the latent variables can focus more on the global information ( e.g. , paragraph-level semantic features) (Chen et al., 2016).
Empirically, we also found that this strategy helps the generative network to make more effective use of the latent variable z (indicated by a larger KL loss term).
3. H IERARCHICALLY - STRUCTURED LATENT VARIABLES
Due to the use of an autoregressive LSTM decoder for word-level predictions, the problem of “posterior collapse”, where the model tends to ignore the latent codes while decoding, may still exist.
In this work, we consider one effective way to mitigate this problem by making less restrictive assumptions regarding the prior distribution of the latent variable.
Speciﬁcally, we propose to leverage a hierarchy of latent variables for text modeling.
As shown in Figure 1, the inference network goes upward through each latent variable to infer their posterior distributions, while the generative network samples downward to obtain the prior distributions over the latent variables.
In this manner, the prior distribution of latent variable at the bottom is inferred from the top-layer latent codes, rather than ﬁxed (as in a standard VAE model).
Consequently, the model is endowed with more ﬂexibility to abstract useful high-level features (Gulrajani et al., 2016), which can then be leveraged by the hierarchical LSTM network.
Without loss of generality, here we choose to employ a two-layer hierarchy of latent variables, where the bottom and top ones are denoted as z 1 and z 2 , respectively.
Note that this framework can be extended easily to more latent-variable layers.
Empirically, we found that the two-layer setup already gives rise to much more informative latent codes, relative to a standard VAE with one stochastic layer.
The posterior distributions over the latent variables are assumed to be conditionally independent given the input x .
Thus, the joint posterior distribution of the two latent variables can be written as:
[Equation]
Concerning the generative network, the latent variable at the bottom is sampled conditioning on the one at the top.
Thus, we have:
[Equation]
To optimize the parameters of the inference and generative networks, the second term in the VAE objective, D KL ( q φ ( z | x ) || p ( z )) , can be regarded as the KL divergence between the joint posterior and prior distributions of the two latent variables.
Thus, the variational lower bound can be written as (given the assumptions in (4) and (5)):
[Equation]
Note that the left-hand side of (7) is the abbreviation of D KL ( q φ ( z 1 , z 2 | x ) || p
( z 1 , z 2 )) .
Given the
Gaussian assumption for both the prior and posterior distributions
[Equation]
both KL divergence terms can be written in closed-form.
E XPERIMENTS 4. E XPERIMENTAL SETUPS
Datasets We evaluate the proposed hierarchically-structured VAE on both generic (unconditional) long text generation and conditional paragraph generation (with additional text input as auxiliary information).
For the former, we employ two datasets: Yelp Reviews (Zhang et al., 2015) and
ArXiv Abstracts (Celikyilmaz et al., 2018).
For the conditional-generation case, we consider the task of synthesizing a paper abstract (which typically includes several sentences) conditioned on the paper title (with the ArXiv Abstract dataset).
We employ a hierarchical CNN architecture as the inference/encoder network.
Speciﬁcally, a sentence-level CNN encoder is ﬁrst applied to each sentence to obtain a ﬁxed-length vector.
Afterwards, a paragraph-level CNN encoder is utilized to aggregate the vectors with respect to all sentences.
Details of the dataset, the experimental setup and model architectures are provided in the Supplementary Materials (SM).
Model speciﬁcation To abstract meaningful representations from the input sentences, we employ a hierarchical CNN architecture as the inference/encoder network.
Speciﬁcally, a sentencelevel CNN encoder is ﬁrst applied to each sentence to obtain a ﬁxed-length vector.
Afterwards, a paragraph-level CNN encoder is utilized to aggregate the vectors with respect to all sentences.
In the single-variable hier -VAE, the paragraph feature vector is fed into linear layers to infer the mean and variance of the latent variable z .
For the double-variable case, the feature vector is further transformed with two MLP layers, and then is used to compute the mean and variance of the top-level latent variable.
The dimension of latent variable z is set to 300 .
The dimensions of both the sentence-level and word-level LSTM decoders are set to 512 .
For the generative networks, to infer the bottom-level latent variable ( i.e. , modeling p ( z 1 | z 2 ) ), we ﬁrst feed the sampled latent codes from z 2 to two MLP layers, which is followed by two linear transformation to infer the mean and variance of z 1 , respectively.
The model is trained using Adam (Kingma & Ba, 2014) with a learning rate of 3 × 10 − 4 for all parameters, with a decay rate of 0.99 for every 3000 iterations.
Dropout (Srivastava et al., 2014) is employed on both word embedding and latent variable layers, with rates selected from { 0.3, 0.5, 0.8 } on the validation set.
We set the mini-batch size to 128.
Following (Bowman et al., 2016)
we adopt the KL cost annealing strategy to stabilize training: the KL cost term is increased linearly to 1 until 10,000 iterations.
All experiments are implemented in Tensorﬂow (Abadi et al., 2016), using one NVIDIA GeForce GTX TITAN X GPU with 12GB memory.
Baselines For the language modeling experiments, we implement several baselines: language model with a ﬂat LSTM decoder ( ﬂat -LM), VAE with a ﬂat LSTM decoder ( ﬂat -VAE), language model with a hierarchical LSTM decoder ( hier -LM).
For generic text generation, we further consider two recently proposed generative models as the baselines: Adversarial Autoencoders (AAE) (Makhzani et al., 2015) and Adversarially-Regularized Autoencoders (ARAE) (Zhao et al., 2018a).
Instead of penalizing the KL divergence term, AAE introduces a discriminator network to match the prior and posterior distributions of the latent variable.
Further, AARE proposes to learn a more ﬂexible prior by assuming no speciﬁc forms of the latent space.
For the proposed hierarchicallystructured VAE, we implement two variants, one with a single latent variable and another with double latent variables, denoted as hier -VAE-S and hier -VAE-D, respectively.
Our code will be released to encourage future research.
4. L ANGUAGE M ODELING R ESULTS
We ﬁrst evaluate our method on language modeling task using Yelp and ArXiv datasets, where we report the negative log likelihood (NLL) and perplexity (PPL).
Following (Bowman et al., 2016;
Yang et al., 2017; Kim et al., 2018), we utilize the KL loss term to measure the extent of “posterior collapse”.
For this experiment we use the following baselines: language model with a ﬂat LSTM decoder, VAE with a ﬂat LSTM decoder, language model with a hierarchical LSTM decoder, VAE with a hierarchical LSTM decoder.
As shown in Figure 2, on the Yelp dataset, the standard textVAE with a ﬂat LSTM decoder has a KL divergence term very close to zero, indicating that the generative model makes neligible use of the information from latent variable z .
Consequently, the ﬂat -VAE model obtains slightly worse NNL and PPL relative to a ﬂat LSTM-based language model.
In contrast, with a hierarchical LSTM decoder, the KL divergence cost term becomes much larger, demonstrating that the VAE model tends to leverage more information from the latent variable in the decoding stage.
Further, the PPL of hier -VAE is also decreased from 47.9 to 46.6 (compared with a LM with hierarchical decoder), further showing that the sampled latent codes has indeed helped make word-level predictions.
[Table]
Moreover, with a hierarchy of latent variables, the hierVAE-D model exhibits an even larger KL divergence cost term (increased from 3 .
6 to 6 . 8 ) than that with a single latent variable, indicating that more information from the latent variable has been utilized by the generative network.
This may be attributed to the fact that the latent variable priors of the hier -VAE-D model are inferred from the data, rather than a ﬁxed standard Gaussian distribution.
As a result, the model is endowed with more ﬂexibility to encode informative semantic features in the latent variables, yet matching their posterior distributions to the corresponding priors.
More importantly, by effectively exploiting the sampled latent codes, hier -VAE-D achieves the best PPL results on both datasets (on ArXiv dataset our hierarchical decoder outperforms the LM by reducing the PPL from 58 .
1 down to 54 .
[Table] [Table]
4. G ENERIC T EXT G ENERATION
We further evaluate our method by examining the quality of generated paragraphs.
In this regard, we randomly sample 1000 latent codes and obtain the generated text sequences by feeding them to the trained generative networks.
We employ corpus-level BLEU score (Papineni et al., 2002) to quantitatively evaluate the generated paragraphs.
Speciﬁcally, we follow the strategy in (Yu et al., 2017; Zhang et al., 2017) and use the entire test set as the reference.
The BLEU scores for thegenerated sentences are averaged to obtain the ﬁnal score for each model.
As shown in Table 5, VAE tends to be a stronger baseline for paragraph generation, exhibiting higher corpus-level BLEU scores than both AAE and ARAE.
This observation is consistent with the results shown in (C ´ ıfka et al., 2018).
The VAE with hierarchical decoder again demonstrates better BLEU scores than the one with a ﬂat decoder, indicating that the plan-ahead mechanism associated with the hierarchical decoding process indeed beneﬁts the sampling quality.
Moreover, hier -VAE-D exhibits slightly better results than hier -VAE-S.
We attribute this to the more ﬂexible prior distribution of hier -VAE-D, which improves the ability of the inference networks to extract semantic features from a paragraph and thus yields more informative latent codes.
Table 3 shows an example generated text from hier -VAE-H and ﬂat -VAE-H.
Compared to the our hierarchical model, using ﬂat decoders with ﬂat VAE architecture exibits n-gram repetitions as well as less variations in the descriptions.
The hieararhical model, on the other hand, contains more information with less repetitions (word or semantic semantic repetitions) indicating a more coherent generated text.
A the service was great, the receptionist was very friendly and the place was clean, we waited for a while, and then our room was ready .
• same with all the other reviews, this place is a good place to eat, i came here with a group of friends for a birthday dinner, we were hungry and decided to try it, we were seated promptly.
• this place is a little bit of a drive from the strip, my husband and i werelookingforaplacetoeat, allthefoodwasgood, theonlything i didn t like was the sweet potato fries.
• this is not a good place to go, the guy at the front desk was rude and unprofessional, it s a very small room, and the place was not clean.
• service was poor, the food is terrible, when i asked for a reﬁll on my drink, no one even acknowledged me, they are so rude and unprofessional.
B how is this place still in business, the staff is rude, no one knows what they are doing, they lost my business .
The Continuity of Latent Space
Following (Bowman et al., 2016), we further measure the continuity of the learned latent space.
Speciﬁcally, two points are randomly sampled from the prior latent space (denoted as A and B , respectively).
Sentences are generated based on the equidistant intermediate points along the linear trajectory between A and B .
As shown in
Table 4, these intermediate samples are all realistic-looking reviews that are syntactically and semantically reasonable, demonstrating the smoothness of the learned VAE latent space.
Interestingly, we even observe that the generated sentences gradually transit from positive to negative sentiment along the linear trajectory.
To validate that the sentences are not generated by simply retrieving the training data, we further ﬁnd the closest instance, among the entire training set, for each generated review.
We demonstrate the details of the results in the supplementary material (Table 14).
Diversity of Generated Paragraphs We also evaluate the diversity of random samples from a trained model, since one model might generate realistic-looking sentences while suffering from severe mode collapse ( i.e. , low diversity).
Three metrics are employed to measure the diversity of generated paragraphs: Self-BLEU scores (Zhu et al., 2018), unique n -grams (Fedus et al., 2018) and entropy score (Zhang et al., 2018).
Speciﬁcally, for a set of sampled sentences, the Self-BLEU
metric calculates the BLEU score of each sample with respect to all other samples as the reference (the numbers over all samples are then averaged); the unique score computes the percentage of
[Table]
[Table]
unique n -grams within all the generated reviews; and the entropy score measures how evenly the empirical n -gram distribution is for a given sentence, which does not depend on the size of testing data, as opposed to unique scores.
Note that all three metrics are the lower, the better.
Here, we randomly sample 1000 reviews from each model, and the corresponding results are shown in Table 6.
Note that a small self-BLEU score must be accompanied with a large BLEU score to justify the effectiveness of a model, i.e. , being able to generate realistic-looking as well as diverse samples.
Among all the VAE variants, hier -VAE-D shows the smallest BLEU score and largest unique n -grams percentage, further demonstrating the advantages of making both the generative networks and latent variables hierarchical.
As for AAE and ARAE, although they exhibit better diversity according to both metrics, their corpus-level BLEU scores are much worse relative to hierVAE-D.
Thus, we leverage human evaluation for further comparison.
Visualization of the learned embeddings To further prove the capability of our model to extract global features, we conducted an additional experiment to visualize the learned latent variable.
Speciﬁcally, from arxiv dataset, we select the most frequent four classes/topics and re-train our hier-VAE-D model on the corresponding abstracts.
The latent codes sampled from the learned inference network are visualized with t -SNE.
As shown in
Figure 2, each point indicates one paper abstract and the color of each point indicates the category it belongs to.
The embeddings of the same label are indeed very close in the 2-D plot, while those with different labels are relatively farther away from each other.
Additionally, the embeddings of the abstracts belonging to the High Energy Physics and Nuclear topics are meshed, which is expected since these two topics are semantically highly related.
Note that the model is trained in a totally unsupervised manner, and the inference network is able to extract meaningful global patterns from the input paragraph.
in this work we study the dynamics of a colloidal suspension of frictionless , the capillary forces are driven by the UNK UNK , when the substrate is a thin ﬁlm , the system is driven by a periodic potential , we also study the dynamics of the interface between the two different types of particles ﬁnding the largest system ﬁnd the value the number states ﬁrst
Human Evaluation We conducted human evaluation using Amazon Mechanical Turk to assess the coherence and non-redundancy of the texts generated from our models in comparison to the baselines, which is difﬁcult to measure based on automated metrics.
Given a pair of generated reviews, the judges are asked to select their preferences (no difference between the two reviews is also an option) according to the following four evaluation criteria: ﬂuency & grammar (G), consistency (C), non-redundancy (N), and overall (O).
Details of the evaluation are provided in the supplementary material.
As shown in Table 8, hier -VAE generates superior human-looking samples compared to ﬂat -VAE on the Yelp Reviews dataset.
Even though both models underperfom when compared against the ground-truth real reviews, hier -VAE was rated higher in comparison to ﬂat -VAE (raters ﬁnd hier -VAE closer to human-generated than the ﬂat -VAE) in all the criteria evaluation criteria.
We further compare our methods against AAE (the same data preprocessing steps and hyperparameters are employed).
The results show that hier -VAE again produces more grammatically-correct and semantically-coherent samples relative to the AAE baseline.
4. C ONDITIONAL P ARAGRAPH G ENERATION
[Table] [Table]
We further evaluate the proposed VAE model on a conditional generation task.
Speciﬁcally, we consider the task of generating abstract of a paper based on the corresponding title.
The same arXiv dataset is utilized [Table] where title and abstract are given as paired text sequences.
The title is used as input of the inference network.
For the generative network, instead of reconstructing the same input ( i
[Table]
e. , title), the paper abstract is employed as the target for decoding.
We compare the hier -VAE-D model against hier -LM.
We observe that the hier -VAED model achieves a test perplexity of 55 .
7 (with a KL term of 2 . 57 ), which is smaller that the test perplexity of hier -LM ( 58 . 1 ).
This indicates that the information from the title has indeed been leveraged by the generative network to facilitate the decoding process.
In table 9 we show two generated samples from the hierVAE-D model.
4. A BLATION S TUDY 4.5. T HE ARCHITECTURE OF ENCODER NETWORKS
[Table]
To investigate the impact of encoder networks on the VAE’s performance, we further conduct an ablation study, where the herarchical
CNN encoder in the hier -VAE model is replaced with a ﬂat CNN encoder or a hierarchical LSTM encoder, respectively. The corresponding results are shown in Table 10. It can be observed that the model with a ﬂat CNN encoder yields worst (largest) perplexity, suggesting that it is beneﬁcial to make the encoder hierarchical. Additionally, hierarchical CNN encoder exhibits slightly better results than hierarchical
LSTM encoder according to our experiments.
4.5. M ITIGATING “ POSTERIOR COLLAPSE ” ISSUE
[Table]
To resolve the “posterior collapse” issue of training textual VAEs, (Park et al., 2018) also introduced a strategy called utterance drop (u.d).
Speciﬁcally, they proposed to weaken the autoregressive power of hierarchical RNNs by dropping the utterance encoder vector with a certain probability.
To investigate the effectiveness of their method relative to our strategy of employing a hierarchy of latent variables, we further conduct a comparative study.
Particularly, we utilize hier-VAE-S as the baseline model and apply the two strategies to it respectively.
The corresponding results on language modeling (Yelp dataset) are shown in Table 11.
Their u.d strategy indeed allows better usage of the latent variable (indicated by a larger KL divergence value).
However, the NLL of the language model becomes even worse, possibly due to the weakening of the decoder during training (similar observations have also been reported in Table 2 of Park et al.
In contrast, our hierarchical prior strategy yields larger KL terms as well as lower NNL value, indicating the advantage of our strategy to mitigate the “posterior collapse” issue.
C ONCLUSION
We introduced a hierarchically-structured variational autoencoder for long text generation.
A hierarchical LSTM generative network is employed, which models the semantic coherence at both the word- and sentence-levels.
As a result, the VAE model is endowed with stronger plan-ahead capability, and the latent variable tends to encode global features that are more useful for the decoding/generating process.
To mitigate the “posterior collapse” issue, a hierarchy of stochastic layers is further utilized, where the priors of the latent variables are learned from the data.
Consequently, more informative latent codes are manifested, indicated by a larger KL loss term yet smaller variational lower bound.
More importantly, the generated samples from the proposed model also exhibit superior quality relative to those from several baseline methods (according to automatic metrics).
Human evaluations further demonstrate that the samples from hier -VAE are less repetitive and more semantically-consistent.
Supplementary Material for “Hierarchically
-Structured Variational Autoencoder for Long Text Generation”
A D ATASETS D ETAILS
In the following, we provide details of dataset preprocessing as well as the experimental setups used in the experiments.
For both Yelp Reviews and ArXiv Abstracts datasets, we truncate the original paragraph to the ﬁrst ﬁve sentences (split by punctuation marks including comma , period and point symbols), where each sentence contains at most 25 words.
Therefore, each paragraph has at most 125 words.
We further remove those sentences that contain of less than 30 words.
The statistics of both datasets are detailed in Table 12.
Note that the average length of paragraphs considered here are much larger than previous generative models for text (Bowman et al., 2016; Yu et al., 2017; Hu et al., 2017; Zhang et al., 2017), since these works considered text sequences that contain only one sentence with at most twenty words.
[Table]
B A DDITIONAL G ENERATED S AMPLES FROM hier -VAE-D VS ﬂat -VAE
We provide additional examples for the comparison between hier -VAE-D vs ﬂat -VAE in Table 13, as a continuation of Table 3.
[Table]
i would give this place zero stars if i could , the guy who was working the front desk was rude and unprofessional , i have to say that i was in the wrong place , and i m not sure what i was thinking , this is not a good place to go to .
this is a great little restaurant in vegas , i had the shrimp scampi and my wife had the shrimp scampi , and my husband had the shrimp scampi , it was delicious , i had the shrimp scampi which was delicious and seasoned perfectly .
my wife and i went to this place for dinner , we were seated immediately , the food was good , i ordered the shrimp and grits , which was the best part of the meal .
very good chinese food, very good chinese food
, the service was very slow, i guess that s what they were doing, very slow to get a quick meal.
we got a gift certiﬁcate from a store, we walked in and were greeted by a young lady who was very helpful and friendly, so we decided to get a cut, I was told that they would be ready in 15 minutes.
we go there for eakfast, i ve been here 3 times and it s always good, the hot dogs are delicious, and the hot dogs are delicious , i ve been there for eakfast and it is so good.
the place was packed, chicken was dry, tasted like a frozen hot chocolate, others were just so so, i wouldn t recommend this place.
do not go here, their food is terrible, they were very slow, in my opinion.
went today with my wife, and received a coupon for a free appetizer, we were not impressed, we both ordered the same thing, and we were not impressed .
the wynn is a great place to eat, the food was great and i had the linguine, and it was so good, i had the linguine and clams,
[Table]
C R ETRIEVED CLOSEST TRAINING INSTANCES OF GENERATED SAMPLES (Y ELP R EVIEWS D ATASET )
We provide samples of retrieved instances from the Yelp Review training dataset which are closest to the generated samples.
Table 14 shows the closest training samples of each generated Yelp review.
The ﬁrst column indicates the intermediate generated sentences produced from linear transition from a point A to another point B in the prior latent space.
The second column on the right are the real sentences retrieved from the training set that are closest to the ones generated on the left (determined by BLEU-2 score).
We can see that the retrieved training data is quite different from the generated samples, indicating that our model is indeed generating samples that it has never seen during training.
[Table]
A the service was great, the receptionist was very friendly and the place was clean, we waited for a while, and then our room was ready .
i ve only been here once myself , and i wasn t impressed , the service was great , staff was very friendly and helpful , we waited for noth- ing • same with all the other reviews, this place is a good place to eat, i came here with a group of friends for a birthday dinner, we were hungry and decided to try it, we were seated promptly.
i really love this place , red robin alone is a good place to eat , but the service here is great too not always easy to ﬁnd , we were seated promptly , ought drinks promptly and our orders were on point .
• this place is a little bit of a drive from the strip, my husband and i were looking for a place to eat, all the food was good, the only thing i didn t like was the sweet potato fries.
after a night of drinking , we were looking for a place to eat , the only place still open was the grad lux , its just like a cheesecake factory , the food was actually pretty good .
• this is not a good place to go, the guy at the front desk was rude and unprofessional, it s a very small room, and the place was not clean.
the food is very good , the margaritas hit the spot , and the service is great , the atmosphere is a little cheesy but overall it s a great place to go .
• service was poor, the food is terrible, when i asked for a reﬁll on my drink, no one even acknowledged me, they are so rude and unpro- fessional.
disliked this place , the hostess was so rude , when i asked for a booth , i got attitude , a major .
B how is this place still in business, the staff is rude, no one knows what they are doing, they lost my business .
[Table]
D H UMAN EVALUATION SETUP AND DETAILS
Some properties of the generated paragraphs, such as (topic) coherence or non-redundancy, can not be easily measured by automated metrics.
Therefore, we further conduct human evaluation based on 100 samples randomly generated by each model (the models are trained on the Yelp Reviews dataset for this evaluation).
We consider ﬂat -VAE, adversarial autoencoders (AAE) and real samples from the test set to compare with our proposed hier -VAE-D model.
The same hyperparameters are employed for the different model variants to ensure fair comparison.
We evaluate the quality of these generated samples with a blind heads-up comparison using Amazon Mechanical Turk.
Given a pair of generated reviews, the judges are asked to select their preferences (“no difference between the two reviews” is also an option) according to the following 4 evaluation criteria: (1) ﬂuency & grammar , the one that is more grammatically correct and ﬂuent; (2) consistency , the one that depicts a sequence of topics and events that is more consistent; (3) non-redundancy , the one that is better at non-redundancy (if a review repeats itself, this can be taken into account); and (4) overall , the one that more effectively communicates reasonable content.
These different criteria help to quantify the impact of the hierarchical structures employed in our model, while the non-redundancy and consistency metrics could be especially correlated with the model’s plan-ahead abilities.
The generated paragraphs are presented to the judges in a random order and they are not told the source of the samples.
Each sample is rated by three judges and the results are averaged across all samples and judges.