AP o- VAE : Text Generation in Hyperbolic Space (2021)

Abstract: Natural language often exhibits inherent hierarchical structure ingrained with complex syntax and semantics. However, most state-of-the-art deep generative models learn embeddings only in Euclidean vector space, without accounting for this structural property of language. In this paper, we investigate text generation in a hyperbolic latent space to learn continuous hierarchical representations. An Adversarial Poincare Variational Autoencoder (APo-VAE) is presented, where both the prior and variational posterior of latent variables are defined over a Poincare ball via wrapped normal distributions. By adopting the primal-dual formulation of Kullback-Leibler divergence, an adversarial learning procedure is introduced to empower robust model training. Extensive experiments in language modeling, unaligned style transfer, and dialog-response generation demonstrate the effectiveness of the proposed APo-VAE model over VAEs in Euclidean latent space, thanks to its superb capabilities in capturing latent language hierarchies in hyperbolic space.

## introduction the variational autoencoder (vae) @xcite ) is a generative model widely applied to language-generation tasks, which propagates latent codes drawn from a simple prior to manifest data samples through a decoder. the generative model is augmented by an inference network, which feeds observed data samples through an encoder to yield a distribution on the corresponding latent codes. since natural language often manifests a latent hierarchical structure, it is desirable for the latent code in a vae to reflect such inherent language structure, so that the generated text can be more natural and expressive. an example of language structure is illustrated in figure 1 , where sentences are organized into a tree structure. in existing vae-based generative models, such structures are not explicitly considered. the latent code often employs a simple gaussian prior, and the posterior is approximated as a gaussian with diagonal covariance matrix. such embeddings assume euclidean structure, which is inadequate in capturing geometric structure illustrated in figure 1. while some variants have been proposed to enrich the prior distributions @xcite @xcite , there is no evidence that structural information in language can be recovered effectively by the model. hyperbolic geometry has recently emerged as an effective method for representation learning from data with hierarchical structure @xcite . informally, hyperbolic space can be considered as a continuous map of trees. for example, a poincaré disk (a hyperbolic space with two dimensions) can represent any tree with arbitrary low distortion @xcite . in euclidean space, however, it is difficult to learn such structural representation even with infinite dimensions @xcite . ## adversarial poincaré vae we first introduce our hyperbolic encoder and decoder, and how to apply reparametrization. we then provide detailed descriptions on model implementation, explaining how the primal-dual form of kl divergence can help stabilize training. finally, we describe how to adopt vampprior (tomczak and welling, 2018) to enhance performance. a summary of our model scheme is provided in figure 2 . ## related work vae for text generation. many vae models have been proposed for text generation, most of which focus on solving the issue of posterior collapse. the most popular strategy is to alter the training dynamics, keeping the encoder away from bad local optima. for example, variants of kl annealing @xcite @xcite dynamically adjust the weight on the kl penalty term as training progresses. lagging vae @xcite aggressively optimizes the encoder before each decoder update, to overcome the imbalanced training issue between the encoder and decoder. alternative strategies have also been proposed based on competing theories or heuristics. δ-vae @xcite tackles this issue by enforcing a minimum kl divergence between the posterior and the prior. @xcite blames mode-collapse on the auto-regressive design of the decoder and advocates alternative architectures. a semi-amortized inference network is considered by @xcite to bridge the amortization gap between log-likelihood and the elbo. recent work has also shown that posterior collapse can be ameliorated by using more expressive priors and variational posteriors other than gaussian. flow-based vae is considered in @xcite to enhance the flexibility of prior distributions. a topic-guided prior is proposed in @xcite to achieve more controllable text generation. @xcite explores implicit sample-based representations, without requiring an explicit density form for the approximate posterior. @xcite considers replacing the gaussian with the spherical von mises-fisher (vmf) distribution. compared to these prior arts, our model features structured representation in hyperbolic space, which not only captures latent hierarchies but also combats posterior collapse. ## experiments we evaluate the proposed model on three tasks: (i) language modeling, (ii) unaligned style transfer, and (iii) dialog-response generation, with quantitative results, human evaluation and qualitative analysis. ## conclusions we present apo-vae, a novel model for text generation in hyperbolic space. our model can learn latent hierarchies in natural language via the use of wrapped normals for the prior. a primal-dual view of kl divergence is adopted for robust model training. extensive experiments on language modeling, text style transfer, and dialog response generation demonstrate the superiority of the model. for future work, we plan to combine apo-vae with the currently prevailing large-scale pre-trained language models.