On the Encoder-Decoder Incompatibility in Variational Text Modeling and Beyond (2020)

Abstract: Variational autoencoders (VAEs) combine latent variables with amortized variational inference, whose optimization usually converges into a trivial local optimum termed posterior collapse, especially in text modeling. By tracking the optimization dynamics, we observe the encoder-decoder incompatibility that leads to poor parameterizations of the data manifold. We argue that the trivial local optimum may be avoided by improving the encoder and decoder parameterizations since the posterior network is part of a transition map between them. To this end, we propose Coupled-VAE, which couples a VAE model with a deterministic autoencoder with the same structure and improves the encoder and decoder parameterizations via encoder weight sharing and decoder signal matching. We apply the proposed Coupled-VAE approach to various VAE models with different regularization, posterior family, decoder structure, and optimization strategy. Experiments on benchmark datasets (i.e., PTB, Yelp, and Yahoo) show consistently improved results in terms of probability estimation and richness of the latent space. We also generalize our method to conditional language modeling and propose Coupled-CVAE, which largely improves the diversity of dialogue generation on the Switchboard dataset.

## introduction the variational autoencoder (vae) @xcite ) is a generative model that combines neural latent variables and amortized variational inference, which is efficient in estimating and sampling from the data distribution. it infers a posterior distribution for each instance with a shared inference network and optimizes the evidence lower bound (elbo) instead of the intractable marginal log-likelihood. given its potential to learn representations from massive text data, there has been much interest in using vae for text modeling @xcite @xcite . prior work has observed that the optimization of vae suffers from the posterior collapse problem, i.e., the posterior becomes nearly identical to the prior and the decoder degenerate into a standard language model @xcite . a widely mentioned explanation is that a strong decoder makes the collapsed posterior a good local optimum of elbo, and existing solutions include weakened decoders @xcite , modified regularization terms @xcite , alternative posterior families @xcite , richer prior distributions @xcite , improved optimization strategies @xcite , and narrowed amortization gaps @xcite . in this paper, we provide a novel perspective for the posterior collapse problem. by comparing the optimization dynamics of vae with deterministic autoencoders (dae), we observe the incompatibility between a poorly optimized encoder and a decoder with too strong expressiveness. from the perspective of differential geometry, we show that this issue indicates poor chart maps from the data manifold to the parameterizations, which makes it difficult to learn a transition map between them. since the posterior network is a part of the transition map, we argue that the posterior collapse would be mitigated with better parameterizations. to this end, we propose the coupled-vae approach, which couples the vae model with a deterministic network with the same structure. for better encoder parameterization, we share the encoder weights between the coupled networks. for better decoder parameterization, we propose a signal matching loss that pushes the stochastic decod-ing signals to the deterministic ones. notably, our approach is model-agnostic since it does not make any assumption on the regularization term, the posterior family, the decoder architecture, or the optimization strategy. experiments on ptb, yelp, and yahoo show that our method consistently improves the performance of various vae models in terms of probability estimation and the richness of the latent space. the generalization to conditional modeling, i.e., coupled-cvae, largely improves the diversity of dialogue generation on the switchboard dataset. our contributions are as follows: we also generalize coupled-vae to conditional modeling and propose coupled-cvae, which largely improves the diversity of dialogue generation on the switchboard dataset. ## encoder-decoder incompatibility in vae for text modeling to understand the posterior collapse problem, we take a deeper look into the training dynamics of vae. we investigate the following questions. how much backpropagated gradient does the encoder receive from reconstruction? how much does it receive from regularization? how much information does the decoder receive from the encoded text? ## coupling variational and deterministic networks based on the above analysis, we argue that posterior collapse could be alleviated by learning chart maps (i.e., ϕ e and ϕ h ) that better parameterize the data manifold. inspired by the chart maps in dae, we propose to couple the vae model with a deterministic network, outlined in figure 3 . modules with a subscript c are deterministic networks that share the structure with those in the stochastic network. sampling is disabled in the deterministic network, e.g., in the case of gaussian posterior, we use the predicted mean vector for later computation. please find details for other posterior families in appendix b. similar to dae, the coupled deterministic network is optimized solely by the coupled reconstruction loss l c rec , which is the same autoregressive cross-entropy loss as l rec . to learn a well-optimized ϕ e , we share the encoder between the stochastic and the deterministic networks, which leverages the rich gradients backpropagated from l c rec . to learn better ϕ h , we propose to guide ϕ h with a well-learned chart map, i.e., the one characterized by decoder c . thus, we introduce a signal matching loss l match that pushes the h to h c . the objective of our approach is one would question the necessity of sharing the structure of the posterior network by resorting to universal approximation @xcite . specifically, a common question is: why not using an mlp as posterior c ? we argue that each structure has a favored distribution of h in r d , so structure sharing facilitates the optimization when we are learning by gradient descent. for example, the latent space learned by planar flows (rezende and mohamed, 2015) has compression and expansion, and vmf-vae @xcite , which is supported on a sphere, may significantly influence the distribution of h in its ambient space r d . ## conclusions in this paper, we observe the encode-decoder incompatibility of vae for text modeling. we bridge the incompatibility and the posterior collapse problem by viewing the encoder and the decoder as two inadequately learned chart maps from the data manifold to the parameterizations, and the posterior network as a part of the transition map between them. we couple the vae model with a deterministic network and improve the parameterizations via encoder weight sharing and decoder signal matching. our approach is model-agnostic and can be applied to a wide range of models in the vae family. experiments on benchmark datasets, i.e., ptb, yelp, and yahoo, show that our approach improves various vae models in terms of probability estimation and the richness of the latent space. we also generalize coupled-vae to conditional language modeling and propose coupled-cvae. results on switchboard show that coupled-cvae largely improves diversity in dialogue generation.