Auto-Encoding Variational B ayes for Inferring Topics and Visualization (2020)

Abstract: Visualization and topic modeling are widely used approaches for text analysis. Traditional visualization methods find low-dimensional representations of documents in the visualization space (typically 2D or 3D) that can be displayed using a scatterplot. In contrast, topic modeling aims to discover topics from text, but for visualization, one needs to perform a post-hoc embedding using dimensionality reduction methods. Recent approaches propose using a generative model to jointly find topics and visualization, allowing the semantics to be infused in the visualization space for a meaningful interpretation. A major challenge that prevents these methods from being used practically is the scalability of their inference algorithms. We present, to the best of our knowledge, the first fast Auto-Encoding Variational Bayes based inference method for jointly inferring topics and visualization. Since our method is black box, it can handle model changes efficiently with little mathematical rederivation effort. We demonstrate the efficiency and effectiveness of our method on real-world large datasets and compare it with existing baselines.

## introduction visualization and topic modeling are important tools in the analysis of text corpora. visualization methods, such as t-sne @xcite , find low-dimensional representations of documents in the visualization space (typically 2d or 3d) that can be displayed using a scatterplot. such visualization is useful for exploratory tasks. however, there is a lack of semantic interpretation as those visualization methods do not extract topics. in contrast, topic modeling aims to discover semantic topics from text, but for visualization, one needs to perform a post-hoc embedding using dimensionality reduction methods. since this pipeline approach may not be ideal, there has been recent interest in jointly inferring topics and visualization using a single generative model @xcite . this joint approach allows the semantics to be infused in the visualization space where users can view documents and their topics. the problem of jointly inferring topics and visualization can be formally stated as follows. problem. let @xmath0. given a number of topics z, and visualization dimension d, we want to find: to solve this problem, plsv (probabilistic latent semantic visualization) is the first model that attempts to tie together all latent variables of topics and visualization (i.e., υ = {x, φ, β}) in a generative model. its tight integration between visualization and the underlying topic model can support applications such as user-driven topic modeling where users can interactively provide feedback to the model @xcite . plsv can also be used as a basic building block when developing new models for other analysis tasks, such as visual comparison of document collections @xcite . relatively less attention has been paid to methods for fast inference of topics and visualization. existing models often use maximum a posteriori (map) estimation with the em algorithm, which is difficult to scale to large datasets. as shown in figure 12 , to run a plsv model of 50 topics via map estimation on a dataset of modest size (e.g., 20 newsgroups), it takes more than 18 hours using a single core. this long running time limits the usability of these visualization methods in practice. in this paper, we aim to propose a fast auto-encoding variational bayes (aevb) based inference method for inferring topics and visualization. aevb (kingma and welling, 2014a) is a black-box variational method which is efficient for inference and learning in latent gaussian models with large datasets. however, to apply the aevb approach to topic models like lda, one needs to deal with problems caused by the dirichlet prior and by posterior collapse @xcite . one of the successful aevb based methods proposed to tackle those challenges for topic models is avitm @xcite . it is not straightforward to apply aevb or avitm to our problem because of two main challenges. first, as reviewed in section 2, plsv models a document's topic distribution using a softmax function over its euclidean distances to topics. it is not clear how to express this nonlinear functional relationship between three categories of latent variables (i.e., topic distribution θ n , document coordinate x n , and topic coordinate φ z ) when applying avitm to visualization. second, aevb has an assumption that latent encodings are identically and independently distributed (i.i.d.) across samples (casale et al., 2018) @xcite . in our case, this assumption works well with latent document coordinates x where each document n is associated with its latent encoding x n in the visualization space. however, for topic coordinates φ and word probabilities β, that assumption is too strong. the reason is that latent encodings of any topic k w.r.t any documents are not independent, but in fact, in our extreme case these latent encodings are similar, i.e., φ (i) @xmath1, for any documents i, j and any topic z. in other words, φ z is shared across documents. the same argument also applies to word probabilities β. to address the first challenge, we propose to model the nonlinear functional relationship between θ n , x n , φ using a normalized radial basis function (rbf) neural network @xcite . in this model, φ @xmath2, i.e., φ are treated as parameters of the rbf network and will be estimated. similarly, we model β as parameters of a linear neural network that is connected to the rbf network to form the decoder in the aevb approach. by treating φ and β as parameters of the decoder, we can solve the second challenge, though it can be seen that our algorithm does not learn their posterior distributions but rather their point estimates. in section 3, we present in detail our proposed method. we focus on plsv model in this work, though the proposed aevb inference method could be easily adapted to other visualization models. we summarize our contributions as follows: ## proposed auto-encoding variational bayes for inferring topics and visualization we represent a document n as a row vector of word counts: w @xmath3. the marginal likelihood of a document is given by: the marginal likelihood of the corpus is p(d) = n @xmath4, φ, β). note that here we treat φ, and β as fixed quantities that are to be estimated. therefore we are working with a non-smoothed plsv where φ, and β are not endowed with a posterior distribution. by treating φ, and β as model parameters, we ensure that they are shared across all documents in the aevb approach. we will consider a fuller bayesian approach to plsv in our future work. as in avitm, we collapse the discrete latent variable z to avoid the difficulty of determining a reparameterization function for it. the rightmost integral in eq. 3 is the marginal likelihood after z is collapsed. we now only consider the true posterior distribution over latent variable x: p (x|w n , γ, φ, β). due to the intractability of eq. 3, it is intractable to compute the posterior. we approximate it by a variational distribution q (x|w n , η) parameterized by η. the variational parameter η is estimated using an inference network as in aevb. we have the following lower bound to the marginal log likelihood (elbo) of a document: since the prior p(x|γ) = normal (0, γi) is a gaussian, we can let the variational posterior q(x|w n , η) be a gaussian with a diagonal covariance matrix: q(x|w n , η) = normal (µ n , σ n ). the kl divergence between two gaussians in eq. 4 can be computed in a closed form as follows @xcite : in eq. 6, the decoding term log p w n |x (l) , φ, β is computed as: in this network, we have z neurons in the hidden layer and φ z is the center vector for neuron z . the rbf function ρ is a non-linear function that depends on the distance x -φ z and w z,z is the influence weight of neuron z on θ nz where z @xmath5,@xmath6. while w z,z can be estimated by optimizing the elbo, we choose to fix it as w z,@xmath7. the parameters of this network are then the center vectors of z neurons that are the coordinates of topics in the visualization space. the rbf function ρ can have different forms, e.g., gaussian: exp(-1 2 r 2 ), inverse quadratic: 1 1+r 2 , or inverse multiquadric: 1 √ 1+r 2 where @xmath8. when ρ is gaussian, eq. 8 reduces to eq. 1 note that this generalization of θ nz is also discussed in @xcite but not in the context of vae inference. since topic coordinates φ z are now the parameters of the rbf network, they can be shared and used by all documents for computing the topic distributions θ (l) n . in the experiments, we will show the performance of plsv with these rbfs using vae inference. linear neural network for computing θ (l) n β . the output of the above normalized rbf network will be the input of a linear neural network to compute θ (l) n β in the decoding term. we treat β as the parameters, i.e., the linear weights w , of the network and it is computed using a softmax over the network weights to ensure the simplex constraint on β: β = σ(w ). the architecture of the whole variational auto-encoder is given in figure 1 . we use batch normalization @xcite to mitigate the posterior collapse issue found in the aevb approach @xcite . final variational objective function. from eqs. 4, 5, 6, 7, we have the following objective function: ## experiments we evaluate the effectiveness and efficiency of our proposed aevb based inference method for visualization and topic modeling both quantitatively and qualitatively. we use four real-world public datasets from different domains including newswire articles, newsgroups posts and academic papers. ## conclusion we propose, to the best of our knowledge, the first fast aevb based inference method for jointly learning topics and visualization. in our approach, we design a decoder that includes a normalized rbf network connected to a linear neural network. these networks are parameterized by topic coordinates and word probabilities, ensuring that they are shared across all documents. due to our method's black box nature, we can quickly experiment with different rbfs with minimal reimplementation effort. our extensive experiments on four real-world large datasets show that plsv-vae runs much faster than plsv-map while gaining better visualization quality and comparable topic coherence.