O N VARIATIONAL LEARNING OF CONTROLLABLE REPRESENTATIONS FOR TEXT WITHOUT SUPERVISION
Anonymous authors
Paper under double-blind review
A BSTRACT
The variational autoencoder (VAE) has found success in modelling the manifold of natural images on certain datasets, allowing meaningful images to be generated while interpolating or extrapolating in the latent code space, but it is unclear whether similar capabilities are feasible for text considering its discrete nature.
In this work, we investigate the reason why unsupervised learning of controllable representations fails for text.
We ﬁnd that traditional sequence VAEs can learn disentangled representations through their latent codes to some extent, but they often fail to properly decode when the latent factor is being manipulated, because the manipulated codes often land in holes or vacant regions in the aggregated posterior latent space, which the decoding network is not trained to process.
Both as a validation of the explanation and as a ﬁx to the problem, we propose to constrain the posterior mean to a learned probability simplex, and performs manipulation within this simplex.
Our proposed method mitigates the latent vacancy problem and achieves the ﬁrst success in unsupervised learning of controllable representations for text.
Empirically, our method signiﬁcantly outperforms unsupervised baselines and is competitive with strong supervised approaches on text style transfer.
Furthermore, when switching the latent factor ( e.g. , topic) during a long sentence generation, our proposed framework can often complete the sentence in a seemingly natural way – a capability that has never been attempted by previous methods.
I NTRODUCTION
High-dimensional data, such as images and text, are often causally generated through the interaction of many complex factors, such as lighting and pose in images or style and content in texts.
Recently,
VAEs and other unsupervised generative models have found successes in modelling the manifold of natural images (Higgins et al., 2017; Kumar et al., 2017; Chen et al., 2016).
These models often discover controllable latent factors that allow manipulation of the images through conditional generation from interpolated or extrapolated latent codes, often with impressive quality.
On the other hand, while various attributes of text such as sentiment and topic can be discovered in an unsupervised way, manipulating the text by changing these learned factors have not been possible with unsupervised generative models to the best of our knowledge.
C´ıfka et al.
; Zhao et al.
(2018) observed that text manipulation is generally more challenging compared to images, and the successes of these models cannot be directly transferred to texts.
Controllable text generation aims at generating realistic text with control over various attributes including sentiment, topic and other high-level properties.
Besides being a scientiﬁc curiosity, the possibility of unsupervised controllable text generation could help in a wide range of application, e.g. , dialogues systems
(Wen et al., 2016).
Existing promising progress (Shen et al., 2017; Fu et al., 2018; Li et al., 2018; Sudhakar et al., 2019) all relies on supervised learning from annotated attributes to generate the text in a controllable fashion.
The high cost of labelling large training corpora with attributes of interest limits the usage of these models, as pre-existing annotations often do not align with some downstream goal.
Even if cheap labels are available, for example, review scores as a proxy for sentiment, the control is limited to the variation deﬁned by the attributes.
In this work, we examine the obstacles that prevent sequence VAEs from performing well in unsupervised controllable text generation.
We empirically discover that manipulating the latent factors for typical semantic variations often leads to latent codes that reside in some low-density region of the aggregated posterior distribution.
In other words, there are vacant regions in the latent code space (Makhzani et al., 2015; Rezende & Viola, 2018) not being considered by the decoding network, at least not at convergence.
As a result, the decoding network is unable to process such manipulated latent codes, yielding unpredictable generation results of low quality.
In order to mitigate the latent vacancy problem, we propose to constrain the posterior mean to a learned probability simplex and only perform manipulation within the probability simplex.
Two regularizers are added to the original objective of VAE.
The ﬁrst enforces an orthogonal structure of the learned probability simplex; the other encourages this simplex to be ﬁlled without holes.
Besides conﬁrming that latent vacancy is indeed a cause of failure in previous sequence VAEs’, it is also the ﬁrst successful attempt towards unsupervised learning of controllable representations for text to the best of our knowledge.
Experimental results on text style transfer show that our approach signiﬁcantly outperforms unsupervised baselines, and is competitive with strong supervised approaches across a wide range of evaluation metrics.
Our proposed framework also enables ﬁner-grained and more ﬂexible control over text generation.
In particular, we can switch the topic in the middle of sentence generation, and the model will often still ﬁnd a way to complete the sentence in a natural way.
B ACKGROUND : V ARIATIONAL A UTOENCODERS
The variational autoencoder (VAE) (Kingma & Welling, 2013) is a generative model deﬁned by a prior p ( zzz ) and a conditional distribution p θθθ ( xxx | zzz ) .
The VAE is trained to optimize a tractable variational lower bound of log p θθθ ( xxx ) :
[Equation]
where q φφφ ( zzz | xxx ) is a variational distribution parameterized by an encoding network with parameters φφφ , and p θθθ ( xxx | zzz ) denotes the decoding network with parameters θθθ .
This objective tries to minimize the reconstruction error to generate the data, and at the same time regularizes q φφφ ( zzz | xxx ) towards the prior p ( zzz ) .
In this paper, p ( zzz ) is chosen as N (0 , I 0 , I 0 , I ) .
For text modelling, the input xxx is some observed text.
Both the encoding and decoding network are usually recurrent neural networks.
Note that during learning, the decoding network p θθθ only learns to decode conditioned on zzz that are sampled from q φφφ ( zzz | xxx ) .
In other words, the decoding network only learns to process zzz sampled from the aggregated posterior distribution q φφφ ( zzz ) =
E xxx ∼ p
d ( xxx ) q φφφ ( zzz | xxx ) , where p d ( xxx ) is the data distribution.
If q φφφ ( zzz ) has regions of low density, there is no guarantee that p θθθ would decode well in such regions.
This is an important intuition that will become central to our analysis in Sec. 3.
L ATENT V ACANCY P REVENTS E FFECTIVE M ANIPULATION
In this section, we take a deeper look into the aggregated posterior latent space of sequence VAE trained on text, and provide justiﬁcation for the alternative solution we propose in Section 4.
3. O BSERVATIONS FROM U NSUPERVISED S ENTIMENT M ANIPULATION
As pointed out by Bowman et al. (2015), one of the motivations to apply VAEs on text is to allow generation of the sentences conditioned on extrinsic features by controlling the latent codes.
Without annotated labels, no previous methods have successfully learned controllable latent factors as mentioned in Sec. 1.
To understand what is missing, we conduct exploratory experiments to use
VAE for unsupervised sentiment manipulation.
We use the Yelp restaurant reviews dataset and the same data split following Li et al.
We train a β -VAE (Higgins et al., 2017) 1 with a latent space of 80 dimensions, an LSTM encoder, and an
LSTM decoder.
Details about this experiment are described in Appendix A.1.
By inspecting the accuracy on the validation set, we ﬁnd that there exists one dimension of latent code achieving higher than 90% sentiment classiﬁcation accuracy by its value alone, while other latent codes get accuracy around 50% .
Further details can be found in Appendix A.2.
It means that this latent dimension is an effective sentiment indicator.
Similar phenomena have been observed in large-scale language models (Radford et al., 2017).
However, the direct inﬂuence on the generative process of the model observed in Radford et al.
(2017) does not apply on the VAE.
When we try to perform sentiment manipulation by modifying this latent dimension 2 , the decoding network fails to generate the desired outputs most of the time, as evidenced by the poor quantitative evaluation in
Table.
1, and poor samples shown in Appendix A.3.
3. L ATENT V ACANCY IN T EXT M ODELLING
One possible reason for the failure is that the decoding network is never trained on codes like the manipulated ones.
This is the case if the aggregated posterior has holes or regions of low density, and the manipulated codes fall into such vacant regions.
Supposing the aggregated posterior latent space possesses a shape as shown in Fig.
the direct manipulated latent codes will fall out of the aggregated posterior latent space for most input samples.
Such latent codes are never seen by the model during training and possess a low density under the aggregated posterior distribution, leading to unpredictable behaviours during decoding.
To verify our hypothesis demonstrated in Fig.
we empirically estimate the density of sentimentmanipulated codes under the aggregated posterior distribution of our trained VAE.
Here, we approximate the data distribution p d ( xxx ) with the empirical distribution over all the training samples.
As a result, the estimated aggregated posterior distribution is a large mixture of Gaussian distribution.
For all 1000 test samples, we move the dimension of code capturing sentiment from µ − 2 σ to µ + 2 σ where µ and σ are the mean and the standard deviation estimated on all the training samples and measure the averaged negative log-likelihood (NLL) under the aggregated posterior distribution.
As depicted in Fig.
3 (a), the NLL plotted in blue dot curve rises sharply when moving away from µ even if there is only one dimension of code is changing, indicating the existence of the vacancy in the aggregated posterior latent space.
In addition, we draw the histogram of all the test samples’
NLL considering their original latent codes and modiﬁed ones in Fig.
3 (b).
The histogram shows that there is a large divergence in NLL between the original latent codes and the modiﬁed ones.
Also, the modiﬁed latent codes have two separate modes, conﬁrming the irregular shape of the aggregated posterior latent space.
In order to resolve this issue, the approach proposed in this work is to constrain the posterior in a way that the manipulation only happens in a learned simplex, as depicted in Fig.
In this constrained subspace, the phenomenom of low density holes of aggregated posterior is signiﬁcantly reduced, as Fig.
3 (a) and (c) empirically show that there is little change in
NLL of original versus modiﬁed codes.
The details of our method is presented in the next section.
M ETHOD 4. O VERVIEW
The experiments conducted in Sec.
3 validates the existence of vacancy in the aggregated posterior latent space.
One potential way to resolve the problem is to better match the aggregated posterior with the prior (Makhzani et al., 2015; Tomczak & Welling, 2017; Zhao et al., 2018).
However, in terms of unsupervised learning of controllable representation for text, these previous methods have not shown successes; Zhao et al.
(2018) only attempted supervised text style transfer, and also reported negative results from the AAE (Makhzani et al., 2015).
Another way to resolve the vacancy issue is to directly enforce that the aggregated posterior itself has no vacant region anywhere where we would like to perform latent code manipulation.
We propose to map the posterior Gaussian mean to a constrained space, more speciﬁcally a learned probability simplex, where we can encourage the constrained latent space to be ﬁlled without vacancy, and perform manipulation to be within this simplex.
As illustrated in Fig.
2, we add an additional mapping function as part of the encoding network which maps the mean of the Gaussian posterior to a constrained space.
Two regularization terms are introduced later to ensure the learned simplex is not degenerate and that this subspace is well ﬁlled.
In addition, we separately model the relevant factors that we wish to control and the irrelevant factors by splitting zzz into two parts, zzz (1) and zzz (2) , following prior work (Bao et al., 2019).
The ﬁrst part captures the relevant factors that are dominant in the data without an inductive bias from external signals, while the second part learns to encode the remaining local information that is useful for reconstructing the source sentences.
As a result, q φφφ ( zzz | xxx ) is decomposed into q φφφ 1 ( zzz (1) | xxx ) q φφφ 2 ( zzz (2) | xxx ) where φφφ = φφφ 1 ∪ φφφ 2 .
With diagonal covariances the KL divergence term in Eq.
1 splits into two separate KL terms.
In practice, we use a MLP encoding network to parametrize zzz (1) with some sentence representations as the input ( e.g. , averaging GloVe embeddings (Pennington et al., 2014) over the input tokens) and a LSTM encoding network to parametrize zzz (2) .
We only constrain the posterior of zzz (1) and zzz (2) is optimized the same way as the traditional VAE.
4. C ONSTRAINING THE P OSTERIOR
We now describe how to map the mean µµµ of the Gaussian posterior for zzz
(1) ∈
R N to a constrained latent space.
We would like to constrain the mean µµµ to have a structure as follows:
[Equation]
where eee i are vectors representing the relevant factors, p i is the proportion of i th relevant factor encoded in zzz (1) and K is a hyperparameter indicating the number of relevant factors to discover.
In other words, the mean of the Gaussian posterior of zzz (1) is constrained to be inside a K -dimension probability simplex in R N whose vertices are represented by the orthogonal basis vectors eee i , i = 1 , . . .
, K .
Given the outputs of the MLP encoder h h h and log σ σ σ 2 , we learn an additional mapping function π which maps hhh to the constrained posterior space, which can be treated as part of the encoding network:
[Equation]
where EEE =
[ eee 1 , . .
. ,eee K ] is a learnable embedding matrix representing the bases, WWW is the learnable weight matrix, and bbb is the learnable bias vector.
As a result, the constrained posterior is parametrized by µµµ and log σσσ 2 as a Gaussian distribution N ( µµµ, diag ( σσσ 2 )) .
With the mapping function alone, the proposed VAE suffers from posterior collapse (Bowman et al., 2015), a well-known problem where the model ignores the latent code zzz during the training.
Further complicating matters is the fact that there is an abundance of signals for predicting the next token in the text, but the signals indicating high-level semantics are quite sparse.
It is thus unlikely that the VAEs can capture useful relevant factors from raw text without collapse.
For these reasons, we enforce orthogonality in the learnt basis vectors as deﬁned in Eq. 2, which introduces a natural recipe to prevent posterior collapse for zzz (1) .
Note that the KL divergence between q φφφ 1 ( zzz (1) | xxx ) and p ( zzz (1) ) is
[Equation]
With orthogonality in the basis vectors, the ﬁrst term in the above equation can be factorized into
[Equation]
To encourage orthogonality in the basis vectors, a regularization term is added to the objective function:
[Equation]
[Equation]
where III is the identity matrix and α is a hyperparamter.
When L REG = 0 , eee (cid:62) i eee i
=
α .
In this case, µµµ (cid:62) µµµ = α (cid:80) i p 2 i reaches its minimum αK when ppp is a uniform distribution.
The proof can be found in Appendix C.
In practice, L REG will quickly decrease to around 0 , ensuring that the KL term will never fully collapse with the structural constraint.
When it comes to controlled generation, one can choose a vertex or any desired point in the probability simplex, as illustrated in Fig. 2.
Note that the constrained posterior also means that the aggregated posterior can never match the isotropic Gaussian prior.
In other word, we achieve good controlled text generation potentially at the cost of poor uncontrolled generation from the prior, but such is not the focus of this current work, and could potentially be resolved by selecting or learning a better prior as in Tomczak & Welling (2017).
4. F ILLING THE C ONSTRAINED S PACE
Constraining the posterior inside a certain space does not guarantee that this space will be ﬁlled after training.
In order to prevent this, we want the probability distribution over the relevant factors ppp to cover as much of the constrained latent space as possible.
We introduce a reconstruction error of the structured latent code in order to push ppp away from a uniform distribution.
For each input sentence, we randomly sample m sentences from the training data as negative samples.
By applying the same encoding process, we get the structured latent code µµµ ( − ) i for each negative sample.
Our goal is to make the raw latent code hhh similar to the restructured latent code µµµ while different from latent codes µµµ ( − ) i of the negative samples, so that ppp is generally different for each input sample.
The structured reconstruction loss is formulated as a margin loss as follows:
[Equation]
Our ﬁnal objective function is deﬁned as follows:
[Equation]
REC .
R ELATED W ORK 5. U NSUPERVISED L EARNING OF D ISENTANGLED R EPRESENTATIONS
Learning disentangled representations is an important step towards better representation learning (Bengio et al., 2013) which can be useful for (semi-)supervised learning of downstream tasks, transfer and few-shot learning (Peters et al., 2017).
VAEs have achieved promising results for unsupervised learning of disentangled representations.
Several variations of VAEs have been proposed to achieve better disentanglement (Higgins et al., 2017; Kumar et al., 2017; Chen et al., 2016;
Razavi et al., 2019).
However, most recent progress in this direction has been restricted to the domain of images.
5. C ONTROLLED T EXT G ENERATION
In order to perform controllable text generation, previous methods either assume annotated attributes or multiple text datasets with different known styles (Hu et al., 2017;
Shen et al., 2017; Zhao et al., 2018; Fu et al., 2018; Li et al., 2018; Sudhakar et al., 2019; Logeswaran et al., 2018; Lample et al., 2018).
The requirement of labelled data largely restricts the capabilities and the applications of these models.
Instead, all our proposed framework needs is raw text without any annotated attribute.
The dominant underlying relevant factors in the given corpus will be discovered and disentangled by our unsupervised method, which can in turn be used for controlled generation.
E XPERIMENTS
To demonstrate the effectiveness of our approach, we compare it to unsupervised baselines with traditional VAEs, considering the density under the aggregated posterior distribution and the performance on sentiment manipulation.
Following evaluation protocols in text style transfer, we also compare our method to strong supervised approaches.
Furthermore, we showcase the ability of ﬁner-grained style discovery and transition possessed by our system, which has not been attempted in the literature.
In this section, our proposed framework is referred as CP-VAE (Constrained Posterior VAE).
Detailed conﬁgurations including the hyperparameters, model architecture, training regimes, and decoding strategy are found in Appendix B.
6. C OMPARISONS WITH U NSUPERVISED B ASELINES
Experimental setup: We use the same experimental setting and dataset as mentioned in Sec. 3.
The 80 D latent code is split into 16 and 64 dimensions for zzz (1) and zzz (2) respectively.
The sentence representations used for zzz (1) is the averaged GloVe embeddings over the input tokens and K is chosen as 3 .
To decide which basis vector corresponds to which sentiment, we sample 10 positive andnegative sentences respectively in the development set, pass them to the encoder, and choose the basis vector with the highest average p i in ppp =
softmax ( WWWhhh + bbb ) , yielding v p as the positive basis and v n as the negative basis.
If v p and v n are chosen to be the same vector, we choose the index with the second highest p i for v p .
To perform sentiment manipulation, we ﬁx zzz (1) to be the chosen basis vector; that is, v p or v n .
Comparisons on density under the aggregated posterior distribution: First, we do linear interpolation between the two discovered basis vectors v p and v n and estimate the averaged NLL under the aggregated posterior distribution the same way as introduced in Sec. 3.
The green solid curve in
Fig.
3 (a) shows that the NLL of CP-VAE is relatively stable for the whole range of the interpolation.
In Fig.
3 (c), the original latent codes and the modiﬁed ones largely overlap with each other.
Both observations validate the effectiveness of CP-VAE in resolving the latent vacancy problem, leading to signiﬁcant improvements on unsupervised sentiment manipulation, as seen later.
Comparsions with metrics on text style transfer: For quantitative evaluation, we adopt automatic evaluation metrics used in text style transfer (Sudhakar et al., 2019) including classiﬁcation accuracy and language
, whose are We also
As shown in Tab. 1, CP-VAE performs signiﬁcantly better than β -VAE in terms of accuracy, BLEU and GLEU.
The lower perplexity of β -VAE is due to mode collapse, which produces very short pivot sentences such as “great !”.
The results match our observations from the experiments on density under the aggregated posterior distribution, conﬁrming that latent vacancy prevents effective manipulation of the latent codes.
We also conduct an ablation study by removing L REG and L S-REC from the objective.
The results demonstrate that both terms are crucial to the success of CP-VAE.
Without L REG , CP-VAE experiences posterior collapse for zzz (1) .
As a result, v p and v n collide with each other, leading to failure in disentangled representation learning.
Since we choose K as 3 , it is convenient to visualize the samples during training with ppp in the learnt probability simplex, as shown in Fig.
We can see that the whole simplex is mostly covered with samples with the help of L S-REC .
Without L S-REC , the decoding network fails to recognize the basis vectors due to the poor coverage of the probability simplex, causing the model to lose most of its transferring ability.
[Table] C OMPARISONS TO S UPERVISED A PPROACHES ON T EXT S TYLE T RANSFER
Experimental setup: We choose two datasets, Yelp and Amazon, used in works (Li et al., 2018;
Sudhakar et al., 2019) on text style transfer which provide human gold-standard references for the test set.
The same train-dev-test splits are used in our experiments.
Two different sentence representations are used in this experiment, averaged GloVe and BERT (Devlin et al., 2018), denoted as CP-G(loVe) and CP-B(ert) respectively.
The remaining settings are as described in the above section.
Compared supervised approaches: On the two datasets, we compare to three adversarially trained models: StyleEmbedding ( SE ) (Fu et al., 2018), MultiDecoder ( MD ) (Fu et al., 2018), CrossAligned ( CA )
(Shen et al., 2017) and two state-of-the-art models based on a “delete, transform, and generate” framework: DeleteAndRetrieve ( D&R ) (Li et al., 2018) and
Blind-GenerativeStyleTransformer
( B-GST )
(Sudhakar et al., 2019).
Evaluation protocols: Four different automatic evaluation metrics are used to measure the different perspectives of the transferring quality, following Sudhakar et al.
To measure transferring ability, we use pre-trained CNN based classiﬁers achieving 98% and 84% accuracies on the test sets of Yelp and Amazon respectively.
To measure content preservation, we use the BLEU (Papineni et al., 2002) score between the transferred sentences and the source sentences.
To measure ﬂuency, we ﬁnetune
OpenAI GPT-
(Radford et al., 2019) with 345 million parameters on the same trainingdev-test split to obtain the perplexity of generated sentences.
The ﬁne-tuned language models achieve perplexities of 26 . 6 and 34 . 5 on the test sets of Yelp and Amazon respectively.
In addition, Sudhakar et al.
(2019) argued that the Generalized Language Evaluation Understanding Metric (GLEU) has a better correlation with the human judgement.
Here, we use the implementation of GLEU 3 provided by Napoles et al.
(2015) to calculate the GLEU score.
Result Analysis: As observed by Li et al.
and Sudhakar et al.
(2019), accuracy, BLEU score and perplexity do not correlate well with human evaluations.
Therefore, it is important to not consider them in isolation.
Tab. 2 shows that our proposed approaches get similar scores on these metrics with human reference sentences on the second row, indicating that the generated sentences of our proposed approaches is reasonable considering the combination of these metrics.
As seen by Sudhakar et al.
and veriﬁed in Sec. 6.1, GLEU strike a balance between target style match and content retention and correlate well with the human evaluations.
From Tab. 2, CP-VAE consistently outperforms the three adversarially trained models on GLEU by a noticeable margin and achieve competitive results as compared to the recent state-of-the-art models.
By checking the samples generated from the models as shown in Tab. 3, B-GST, the current state-of-the-art, is more [Table] [Table] consistent to the source sentence, which can be expected, since it only makes necessary edits to ﬂip the sentiment.
CP-VAE tends to generate more diverse contents which may not be relevant sometimes, but the overall quality is reasonable considering it is trained without the label information.
More samples can be found in Appendix E
[Table]
6. F INER - GRAINED S TYLE D ISCOVERY AND T RANSITION
[Table]
[Table] World to Sci/Tech [Table]
NEW YORK
[Table]
Federer, the world’s No. 1 player, will miss the rest of
[Table]
season because
[Table]
quadriceps
[Table]
NEW YORK [Table] Federer, the world’s No. 1 player, will miss the rest of the year because of a bid-rigging scandal . To further explore the potential of CP-VAE, we conduct the following exploratory experiments.
We use the AG news dataset constructed by (Zhang et al., 2015), which contains four topic categories which are World, Sports, Business and Sci/Tech , with the title and description ﬁelds.
Here, we drop the title and just use the description ﬁeld to train CP-VAE and set K = 10 .
All four topics are automatically discovered by CP-VAE and identiﬁed as described in Sec. 6.1.
We also compare the results of our identiﬁed topics to standard baselines for unsupervised topic modelling, the details can be found in Appendix D.
We choose a basis vector discovered by our model and generate a few tokens.
Then, we switch the basis vector and continue the generation until the end-of-seq token is generated.
Generated samples are shown in Table 4.
We see that our model learns to transition from one topic to another in a natural and ﬂuent way within the same sentence.
Several observations can be made based on these samples: (1) it is good at detecting name entities and replacing them with the name entities related to the chosen topic; (2) there is no hard restriction on when to switch the topic; the model will determine an appropriate way to do the transition by itself.
Such observations conﬁrm that CP-VAE possesses a ﬁlled constrained latent space which make the latent code robust to manipulation across different time steps, which can be effectively reﬂected in the generation process.
Due to space limitations, we put more samples in Appendix F.
C ONCLUSION
In this work we investigate latent vacancy as an important problem in unsupervised learning of controllable representations when modelling text with VAEs.
To mitigate this we propose to constrain the posterior within a learned probability simplex, achieving the ﬁrst success towards controlled text generation without supervision.
A D ETAILS ABOUT E XPLORATORY E XPERIMENTS A. M ODEL D ETAILS
For the β -VAE we used for the exploratory experiments, we use a LSTM encoding network and a
LSTM decoding network.
For the encoding network, the input size is 256, and the hidden size is 1,024.
For the decoding network, the input size is 256, the hidden size is 1,024, and dropouts with probability 0.5 are applied on after the embedding layer and the LSTM layer in the decoding network.
β is chosen as 0.35, the dimension for the latent code is 80, and the batch size is 32.
We use SGD with learning rate 1.0 to update the parameters for both the encoding and the decoding network.
We train the model until the reconstruction loss stops decreasing.
A. I DENTIFYING THE LATENT FACTOR INDICATING THE SENTIMENT
First, we normalize the value of each latent code by subtracting the mean estimated over all the training samples.
Then we use the polarity of each latent code to classify the sentiment in the validation set.
The one with the highest accuracy is identiﬁed as the latent factor indicating the sentiment.
A. S AMPLES GENERATED FROM
[Table]
A. M ANIPULATION S TRATEGIES
Following manipulation strategies have been attempted: (1) ﬁxing the relevant factor to µ + 2 σ and µ − 2 σ ; (2) ﬁxing the relevant factor to µ − σ and µ − σ ; (3) ﬁxing the relevant factor to the maximum value and the minimum value of the relevant factor appearing in the training samples; (4) calculating a latent vector based on 10 manually constructed parallel sentences with opposite sentiment while keeping other factors unchanged.
However, none of these four strategies is effective considering the generation results.
We report the result with the ﬁrst strategy in the paper, since it performs the best considering the accuracy and the BLEU score.
B D ETAILS ABOUT E XPERIMENTS ON T EXT S TYLE T RANSFER B. T RAINING R EGIMES
Across all the datasets, we use Adam with learning rate 0.001 to update the parameters for the encoding network, while SGD with learning rate 1.0 to update the parameters for the decoding network.
The batch size is chosen to be 32.
Dropouts with drop probability 0.5 are applied on applied on after the embedding layer and the LSTM layer in the decoding network.
We train the model until the reconstruction loss stops decreasing.
B. M ITIGATING P OSTERIOR C OLLAPSE
For the structured part zzz (1) , we use β -VAE setting β as 0.2 across all the datasets.
For the unstructured part zzz (2) , different strategies are employed for each dataset:
• Yelp : β -VAE setting β as 0.35.
• Amazon : β -VAE setting β as 0.35.
• AG-News : KL annealing, from 0.1 to 1.0 in 10 epochs.
B. H YPERPARAMETER S ETTINGS
[Table]
The hyperparameters are chosen by checking L VAE , KL, and the generated outputs on the development set for Yelp and AG-News .
Amazon follows the same setting as Yelp without extra tuning.
B. D ECODING S TRATEGY
For decoding, we use beam search with a beam size of 5.
C P ROOF OF M INIMALIZATION OF E Q .
The problem can be formulated as an optimization problem as follows: maximize
K (cid:88) i
= p 2 i , subject to
K (cid:88) i = p i = 1 .
By introducing a Lagrange multiplier λ , the Lagrange function is deﬁned as
L ( p 1 , p 2 , . .
. , p K , λ ) =
K (cid:88) i
= p 2 i − λ (
K (cid:88) i
= p
i − 1) .
In order to ﬁnd the optimal point, we require that ∂
∂p i (cid:32) K (cid:88) i
= p 2 i − λ (
K (cid:88) i
= p
i − 1) (cid:33) =
2 p i − λ = 0 , i = 1 , 2 , . .
, K, which shows that all p i are equal.
By using the constraint (cid:80) i p i = 1 , we ﬁnd p i =
1 K , i = 1 , 2 , . . .
, K .
By plugging into the results, µµµ (cid:62) µµµ = α (cid:80) i p 2 i reaches its minimum αK .
D C OMPARISONS WITH B ASELINES ON T OPIC M ODELLING
Experimental setup: We use the AG news dataset for this task constructed by (Zhang et al., 2015).
It contains four topic categories which are World, Sports, Business and Sci/Tech , with the title and description ﬁelds.
For each category, there are 30 , 000 training samples and 1 , 900 test samples.
In this paper, we drop the title and just use the description ﬁeld.
We compare our approach to two standard baselines for unsupervised topic modelling: (1) LDA (Blei et al., 2003), a standard implementation of LDA is used for this baseline 4 ; (2) k -means .
To show the power of our approach beyond the pre-trained sentence representations, we perform k -means clustering directly on the sentence representations.
Following (Manning et al., 2010), we assign each inferred topic to one of the gold-standard topics with the optimal mapping and report the precision ( a.k.a. purity), recall ( a.k.a. collocation) and F 1 score.
The number of topics is chosen to be 10 .
The results reported for the baselines and our model are the average over 10 runs.
Quantitative results: The results are shown in Table 7.
We can see that our approach achieves comparable results to LDA while signiﬁcantly outperforming k -means in all four categories, indicating that our approach can go beyond just clustering on pre-trained sentence representations.
[Table] [Table]
SRC this was the best i have ever had !
B-GST this was the worst place i have ever had !
CP-G this was the worst pizza i have ever had !
CP-B this was the worst i have ever had !
SRC friendly and welcoming with a fun atmosphere and terriﬁc food .
B-GST the hummus is ridiculously bland and bland .
CP-G rude and unorganized with a terrible atmosphere and coffee .
CP-B the hummus is ridiculously greasy and tasteless .
SRC i ordered the carne asada steak and it was cooked perfectly !
B-GST i ordered the carne asada steak and it was just as bad !
CP-G i ordered the carne asada steak and it was n’t cooked and it was lacking .
CP-B i ordered the carne asada burrito and it was mediocre .
SRC the owner is a hoot and the facility is very accommodating .
B-GST the owner is a jerk and the facility is very outdated .
CP-G the owner is a hoot and the facility is empty and the layout is empty .
CP-B the owner is a riot and the facility is very clean.
SRC i will be going back and enjoying this great place !
B-GST i wo n’t be going back and this place is horrible !
CP-G i will be going back and eat this pizza hut elsewhere .
CP-B i will be going back and hated the worst dining experience .
[Table]
there is deﬁnitely not enough room in that part of the venue .
B
[Table]
GST there is plenty enough seating in that part of the venue .
CP
[Table]
G there is deﬁnitely an authentic dinner in that part .
CP-B there is deﬁnitely a nice theatre in that part .
SRC but it probably sucks too !
B-GST
but it tastes great too !
CP-G
but it ’s truly fun and insanely delicious .
CP-B
but it ’s probably wonderful when u !
SRC always rude in their tone and always have shitty customer service !
B-GST always in tune with their tone and have great customer service .
CP-G always great with their birthdays and always excellent music .
CP-B always accommodating and my dog is always on family .
SRC i was very sick the night after .
B-GST i was very happy the night after .
CP-G i was very pleased with the night . CP-B i was very happy with the night .
SRC this is a horrible venue .
B-GST this is a wonderful venue .
CP-G this is a great place for celebrating friends .
CP-B this is a great place for beginners .
E. S ENTIMENT M ANIPULATION ON A MAZON D ATASET
[Table]
[Table]
ve seen are much better .
[Table]
wheels
[Table]
i ve seen are much more good and are much quality .
[Table]
most pizza wheels that i ve seen are much better than are much better
[Table]
however , this is an example of how rosle got it right .
[Table]
however , this game is an example of how rosle loves it .
[Table]
however , this is an example of how toxic . . .
sad . .
obviously .
[Table]
however , this is an example of how cheap .
similar .
cheap advice . cheap advice .
similar .
[Table]
auto shut off after num num hours , which is a good feature .
[Table]
auto shuts off after num num hours , which is a shame .
[Table]
G whipped mask off after num num hours , which is slimy , which is disgusting .
[Table]
auto shut off after num num hours, which is a stupid idea , which seems to be bad .
[Table]
that said , the mic did pic up everything it could .
B-GST that said
, the game took up everything it could .
[Table]
G that said, the shampoo did nt smell him well .
stopped cleaning everything .
ended up smelling sick
[Table]
B that said, the mic did not ﬁt everything on well , let me down it weren t cleaning SRC
i also prefered tha blade weight and thickness of the wustof !
[Table]
GST i also like the blade weight and of the wustof .
[Table]
G i also disliked the blade weight and thickness of the materials .
CP
[Table]
B i also slammed the blade weight and thickness of the wide .
[Table]
the quality is declined quickly by heat exposure .
[Table]
the water is quickly drained by head exposure .
[Table]
G the quality is utilitarian so grinding or sandwiches .
[Table]
B the quality is priceless quickly by heat rises .
SRC the directions were easy to follow but the quality of the easel was pathetic .
B-GST the directions were easy to follow but the quality of the product was excellent .
CP-G the directions were easy to follow but the quality is good for the quality and is CP-B the directions were easy to follow but the quality is what the quality is like the best quality of SRC multiplayer is just as bad, though thankfully not worse .
B-GST quality is just as good , though thankfully not perfect .
CP-G besides it is just good , though .
those . .
usually . .
usually . . .
CP-B multiplayer is just as bad, though somebody s also so far not so far but no problem .
SRC another energy product that simply wastes our money .
B-GST another energy product that simply saves our money .
CP-G another energy product that simply glides your pasta .
CP-B another energy product that simply wastes this money .
SRC i received the wrong color and it shreds easily .
B-GST i received the color and it works easily .
CP-G i low the new color and it closes easily .
CP-B i received the wrong color and it pours easily from dishwasher and dries easily on garlic easily .
F T EXT T RANSITION E XAMPLES ON AG NEWS
World throughout
BAGHDAD (Reuters) - Iraq ’s interim prime minister , Iyad Allawi , said on Monday that the United States had no intention of withdrawing from the country to end the violence in Iraq .
World to Sports
BAGHDAD (Reuters) - Iraq ’s interim prime minister , Iyad Allawi , said on Monday that the United States had no intention of withdrawing its troops from the country to the end of the year .
World to Business
BAGHDAD (Reuters) - Iraq ’s interim prime minister , Iyad Allawi , said on Monday that the United States had no intention of withdrawing its troops from the country to the country .
World to Sci/Tech
BAGHDAD (Reuters) - Iraq ’s interim prime minister , Iyad Allawi , said on Monday that the United States had no intention of withdrawing its uranium enrichment program to the United States .
Sports throughout
For the ﬁrst time in four years , the US men ’s basketball team won the gold medal in the men ’s 400-meter medley relay .
Sports to World
For the ﬁrst time in four years , the US men ’s basketball team won the gold medal at the Athens Olympics in Athens , where the United
States and the United States have agreed to a peace deal .
Sports to Business
For the ﬁrst time in four years , the US men ’s basketball team won the gold medal at the Athens Olympics on Wednesday , with a surge in crude oil prices .
Sports to Sci/Tech
For the ﬁrst time in four years , the US men ’s basketball team won the gold medal in the men ’s Olympic basketball tournament in Beijing on Tuesday .
Business throughout
NEW YORK (Reuters) - U.S. stocks opened higher on Friday , as oil prices climbed above $48 a barrel and the Federal Reserve raised interest rates by a quarter percentage point .
Business to World
NEW YORK (Reuters) - U.S. stocks opened higher on Friday , as oil prices climbed above $48 a barrel and the Federal Reserve raised interest rates by a quarter percentage point .
Business to Sports
NEW YORK (Reuters) - U.S. stocks opened higher on Friday , as oil prices climbed above $48 a barrel and the Federal Reserve raised interest rates by a quarter percentage point .
Business to Sci/Tech
NEW YORK (Reuters) - U.S. stocks opened higher on Friday , as oil prices climbed above $48 a barrel and the Federal Communications
Commission said it would allow the companies to use mobile phones .
Sci/Tech throughout
SINGAPORE (Reuters) - South Korea ’s Hynix Semiconductor Inc. said on Tuesday it had developed a prototype micro fuel cell recharger for a range of security vulnerabilities in India .
Sci/Tech to World
SINGAPORE (Reuters) - South Korea ’s Hynix Semiconductor Inc. said on Tuesday it had developed a prototype micro fuel cell aimed at ending a standoff with North Korea .
Sci/Tech to Sports
SINGAPORE (Reuters) - South Korea ’s Hynix Semiconductor Inc. said on Tuesday it had developed a prototype micro fuel cell aimed at protecting the world ’s biggest gold medal .
Sci
/Tech to Business
SINGAPORE (Reuters) - South Korea ’s Hynix Semiconductor Inc. said on Tuesday it had developed a prototype micro fuel cell aimed at protecting the world ’s largest oil producer .