Paper:
INFERENCE & INTROSPECTION IN DEEP GENERATIVE MODELS OF SPARSE DATA
Deep generative models such as deep latent Gaussian models (DLGMs) are powerful and popular density estimators. However, they have been applied almost exclusively to dense data such as images; DLGMs are rarely applied to sparse, high-dimensional integer data such as word counts or product ratings. One reason is that the standard training procedures find poor local optima when applied to such data. We propose two techniques that alleviate this problem, significantly improving our ability to fit DLGMs to sparse, high-dimensional data. Having fit these models, we are faced with another challenge: how to use and interpret the representation that we have learned? To that end, we propose a method that extracts distributed representations of features via a simple linearization of the model.
1 INTRODUCTION
Deep latent Gaussian models (DLGMs, a.k.a. variational autoencoders; Rezende et al., 2014; Kingma et al., 2014) have led a resurgence in the use of deep generative models for density estimation. DLGMs assume that observed vectors x are generated by applying a nonlinear transformation (defined by a neural network with parameters θ) to a vector of Gaussian random variables z. Learning in DLGMs proceeds by approximately maximizing the average marginal likelihood p(x) ≡∫ z p(z)p(x|z)dz of the observations x. Computing the true marginal likelihood is intractable, so we resort to variational expectation-maximization (Bishop, 2006), an approximation to maximumlikelihood estimation. To learn the parameters θ of the generative model, the procedure needs to find a distribution q(z|x) that approximates the posterior distribution p(z|x) of the latent vector z given the observations x. In the past, such q distributions were fit using iterative optimization procedures (e.g., Hoffman et al., 2013). But Rezende et al. (2014) and Kingma et al. (2014) showed that q(z|x) can be parameterized by a feedforward “inference network” with parameters φ, speeding up learning. This inference network is trained jointly with the generative model; as training proceeds, the inference network learns to approximate posterior inference on the generative model, and the generative model improves itself using the output of the inference network.
Embedded within this procedure, however, lies a potential problem: both the inference network and the generative model are initialized randomly. Early on in learning, the inference network’s q(z|x) distributions will be poor approximations to the true posterior p(z|x), and the gradients used to update the parameters of the generative model will therefore be poor approximations to the gradients of the true log-likelihood log p(x). Previous stochastic variational inference methods (Hoffman et al., 2013) were slower, but suffered less from this problem since for every data-point, a set of variational parameters was optimized within the inner loop of learning. In this work, we investigate blending the two methodologies for learning models of sparse data. In particular, we use the parameters predicted by the inference network as an initialization and optimize them further during learning. When modeling high-dimensional sparse data, we show that updating the local variational parameters yields generative models with better held-out likelihood, particularly for deeper generative models.
What purpose is served by fitting bigger, deeper, more powerful generative models? Breiman (2001) argues that statistical discriminative modeling falls into two schools of thought: the data modeling culture and the algorithmic modeling culture. The former advocates the use of predictive models that assume interpretable, mechanistic processes while the latter advocates the use of black box techniques with an emphasis on prediction accuracy. Breiman’s arguments also ring true about the
z
x θ φ Figure 1: Deep Latent Gaussian Model: The Bayesian network depicted here comprises a single latent variable with the conditional probability p(x|z) defined by a deep neural network with parameter θ. The dotted line represents the inference network parameterized by φ, which is used for posterior inference at train and test time.
divide between deep generative models with complex conditional distributions and simpler, more interpretable statistical models. Consider a classic model such as Latent Dirichlet Allocation (Blei et al., 2003). It is outperformed in held-out likelihood (Miao et al., 2016) by deeper generative models and assumes a simple probabilistic process for data generation that is unlikely to hold in reality. Yet, its generative semantics lend it a distinct advantage: interpretability. The word-topic matrix in the model allows practitioners to read off what the model has learned about the data. Is there a natural way to interpret the generative model when the conditional distributions are parameterized by a deep neural network?
Our second contribution is to introduce a simple, easy to implement method to interpret what is being learned by generative models such as DLGMs whose conditional probabilities are parameterized by deep neural networks. Our hope is to narrow the perceived gulf between a complex generative model’s representational power and its interpretability. We use the Jacobian of the conditional distribution with respect to latent variables in the Bayesian network to form embeddings (or Jacobian vectors) of the observations. We investigate the properties of the Jacobian vectors obtained from deeper, more non-linear generative models.
2 BACKGROUND
Generative Model: We consider learning in generative models of the form shown in Figure 1. We observe a set of D word count vectors x1:D, where xdv denotes the number of times that word index v ∈ {1, . . . , V } appears in document d. We assume we are given the total number of words per document Nd ≡ ∑ v xdv , and that xd was generated via the following generative process: zd ∼ N (0, I); γ(zd) ≡ MLP(zd; θ); µ(zd) ≡ exp{γ(zd)}∑ v exp{γ(zd)v} ; xd ∼ Multinomial(µ(zd), Nd).
(1)
That is, we draw a Gaussian random vector, pass it through a multilayer perceptron (MLP) with parameters θ, pass the resulting vector through the softmax (a.k.a. multinomial logistic) function, and sample Nd times from the resulting distribution over the vocabulary.1
Variational Learning: For ease of exposition notation we drop the subscript on xd to form x referring to a single data point. We need to approximate the intractable posterior distribution p(z|x) during learning. Using the well-known variational principle, we can obtain the lower bound on the log marginal likelihood of the data (or L(x; θ, φ)) in Eq. 2. where the inequality is by Jensen’s inequality.
log p(x; θ) ≥ E qφ(z|x) [log pθ(x|z))]−KL( qφ(z|x)||p(z) ) = L(x; θ, φ), (2)
We leverage an inference network or recognition network (Hinton et al., 1995), a neural network which approximates the intractable posterior, during learning. This is a parametric conditional distribution that is optimized to perform inference. Kingma & Welling (2014); Rezende et al. (2014) use a neural net (with parameters φ) to parameterize qφ(z|x). The challenge in the resulting optimization problem is that the lower bound (2) includes an expectation w.r.t. qφ(z|x), which implicitly depends on the network parameters φ. This difficulty is overcome by using stochastic backpropagation.
With a normal distribution as our variational approximation we have that qφ(z|x) ∼ N (µφ(x),Σφ(x)). µφ(x),Σφ(x) are functions of the observation x, and we denote by ψ(x) :=
1In keeping with common practice, we neglect the multinomial base measure term N ! x1!···xV ! , which amounts to assuming that the words are observed in a particular order.
{µφ(x),Σφ(x)} the local variational parameters predicted by the inference network. A simple transformation allows one to obtain unbiased Monte Carlo estimates of the gradients of Eqφ(z|x) [log pθ(x|z))] with respect to φ. If we assume the prior p(z) is also normally distributed, the KL and its gradients may be obtained analytically. Throughout this paper we will use θ to denote the parameters of the generative model, and φ to denote the parameters of the inference network.
3 METHODOLOGY
Inference with Global Information: Sparse data typically exhibits long tails and learning in the presence of rare features is challenging. Inference networks learn to regress to the optimal posterior parameters for every data point and global information about the relative frequencies of the individual features in the training distribution may present valuable information during learning.
The simplest way to incorporate first order statistics across the training data into the inferential process is to condition on tf-idf (Baeza-Yates et al., 1999) features instead of the raw-counts. tf-idf is one of the most widely used techniques in information retrieval. In the context of building bag-of-words representations for documents, tf-idf re-weight features to increase the influence of rarer words while decreasing the influence of common words appearing in all documents. The tf-idf-transformed word-count vector is x̃dv ≡ xdv log D∑
d′ min{xd′v,1} . After applying transform, the resulting vector x̃
is normalized by its L2 norm. It’s worthwhile to note that leveraging first-order statistics for inference is difficult in the traditional paradigm of tracking variational parameters for each data point but is easy with inference networks.
Optimizing Local Variational Parameters: The inference network initially comprises a randomly initialized neural network. The predictions of the inference network early in optimization are suboptimal variational parameters used to derive gradients of the parameters of the generative model. This induces noise and bias to the gradients used to update the parameters of the generative model; this noise and bias may push the generative model towards a poor local optimum. Previous work has suggested that deep neural networks (which form the conditional probability distributions pθ(x|z)) are sensitive to initialization (Glorot & Bengio, 2010; Larochelle et al., 2009).
To avoid these issues, we only use the local variational parameters ψ(x) predicted by the inference network to initialize an iterative optimizer that maximizes the ELBO with respect to ψ; we use the optimized variational parameters ˆψ(x) to derive gradients for the generative model. We then train the inference network using stochastic backpropagation and gradient descent, holding the parameters of the generative model θ fixed. Our procedure is detailed in Algorithm 1.
Algorithm 1 Pseudocode for Learning: We evaluate expectations in L(x) (see Eq. 2) using a single sample from the variational distribution and aggregate gradients across mini-batches. M = 1 corresponds to performing no additional optimization to the variational parameters We update θ, ψ(x), φ using stochastic gradient descent with adaptive learning rates ηθ, ηψ(x), ηφ obtained via ADAM (Kingma & Ba, 2015)
Inputs: Dataset D := [x1, . . . , xD], Inference Model: qφ(z|x), Generative Model: pθ(x|z), p(z) while notConverged() do
1. Sample datapoint: x ∼ D 2. Estimate local variational parameters ψ(x)1 using qφ(z|x) 3. Estimate ψ(x)M ≈ ˆψ(x) = arg maxψ(x) L(x; θ;ψ(x)) via SGD as:
m = 1, . . . ,M , ψ(x)m+1 = ψ(x)m + ηψ(x) ∂L(x;θ,ψ(x)m)
∂ψ(x)m
4. Update θ as: θ ← θ + ηθ∇θL(x; θ, ψ(x)M ) 5. Update φ as: φ← φ+ ηφ∇φL(x; θ, ψ(x))
end while
Introspection: Linear models are inherently interpretable. Consider linear regression, factor analysis (Spearman, 1904), and latent Dirichlet allocation (LDA; Blei et al., 2003), which (standardizing notation) assume the following relationships:
Regression: E[y|x] = Wx+ b; Factor Analysis: x ∼ N (0, I); E[y|x] = Wx+ b; Latent Dirichlet Allocation: x ∼ Dirichlet(α); E[y|x] = Wx. (3)
In each case, we need only inspect the parameter matrix W to answer the question “what happens to y if we increase xk a little?” The answer is clear—y moves in the direction of the kth row of W . We can ask this question differently and get the same answer: “what is the derivative ∂E[y|x]∂x ?” The answer is simply the parameter matrix W .
For models as in Fig 1, the variability in the training data is assumed to be due to the single latent state z. The relationship between latent variables z and observations x cannot be quickly read off of the parameters θ. But we can still ask what happens if we perturb z by some small dz—this is simply the directional derivative ∂E[x|z]∂z dz. We can interpret this Jacobian matrix in much the same way we would a factor loading matrix, with two main differences. First, the Jacobian matrix ∂E[x|z] ∂z varies with z—the interpretation of z may change significantly depending on context. Second, DLGMs exhibit rotational symmetry—the prior on z is rotationally symmetric, and the MLP can apply arbitrary rotations to z before applying any nonlinearities, so a priori there is no “natural” set of basis vectors for z. For a given Jacobian matrix, however, we can find the most significant directions via a singular value decomposition (SVD).
Jacobian Vectors: We present our method to generate embeddings from Bayesian networks of the form Figure 1. We consider three variants of Jacobian embedding vectors, based on the unnormalized potentials from the MLP, logarithmic probabilities, and linear probabilities respectively:
J (z)pot = ∂γ(z) ∂z
J (z)log = ∂ logµ(z) ∂z
J (z)prob = ∂µ(z) ∂z
(4)
For any z, {J (z)log,J (z)pot,J (z)prob} ∈ RV×K where K is the latent dimension and V is the dimensionality of the observations. It is this matrix that we use to form embeddings. We denote by ui the Jacobian vector obtained from the row of the Jacobian matrix. When not referring to a particular variant, we use J (z) to denote the Jacobian matrix. J (z) is a function of z leaving open the choice of where to evaluate this function. The semantics of our generative model suggest a natural choice: Jmean := Ep(z)[J (z)]. This set of embeddings captures the variation in the output distribution with respect to the latent state across the prior distribution of the generative model. Additionally, one may also evaluate the Jacobian at the approximate posterior corresponding to an observation x. We study how this may be used to obtain contextual word-vectors.
In frameworks that support automatic differentiation (e.g., Theano; Theano Development Team, 2016), J (z) is readily available and we estimate Jmean via Monte-Carlo sampling from the prior. Deriving Jacobian Vectors: For simplicity, we derive the functional form of the Jacobian in a linear model i.e where γ(zd) = Wzd (c.f Eq 1). We drop the subscript d and denote by γi(z), the ith element of the vector γ(z).
p(xi = 1|z) = exp(γi(z))∑ j exp(γj(z))
and γi(z) = wTi z
For linear models, ∇zγi(z) = wi directly corresponds to J (z)pot. Noting that ∇z exp(γi(z)) = exp(γi(z))∇zγi(z) and ∇z ∑ j exp(γj(z)) = ∑ j exp(γj(z))∇zγj(z), we estimate J (z)prob as:
∇zp(xi = 1|z) = ∇z exp(γi(z))∑ j exp(γj(z)) =
∑ j exp(γj(z))∇z exp(γi(z))− exp(γi(z))∇z ∑ j exp(γj(z))
( ∑ j exp(γj(z))) 2
=
∑ j exp(γj(z)) exp(γi(z))wi − exp(γi(z)) ∑ j exp(γj(z))wj
( ∑ j exp(γj(z))) 2
= p(xi = 1|z)wi − p(xi = 1|z) ∑ j p(xj = 1|z)wj
= p(xi = 1|z)(wi − ∑ j p(xj = 1|z)wj)
Similarly, we may compute J (z)log: ∇z log p(xi = 1|z) = wi − ∑ j p(xj = 1|z)wj = ∑ j p(xj = 1|z)(wi − wj) (5)
Denoting a word-pair vector as wi −wj , where wi, wj are columns of the matrix W . If we define the set of all word-pair vectors as S , then Eq 5 captures the idea that the vector representation for a word i lies in the convex hull of S . Furthermore, the word vector’s location in CONV(S) is determined by the likelihood of the pairing word (xj) under the model p(xj = 1|z).
When we use a non-linear conditional probability distribution J (z)log becomes: ∇z log p(xi = 1|z) = ∑ j p(xj = 1|z)(∇zγi(z)−∇zγj(z)) where ∇zγi(z) is a non-linear function of z. To the best of our knowledge, Jacobian Vectors and their properties have not been studied.
4 RELATED WORK
Learning in Deep Generative Models: Salakhutdinov & Larochelle (2010) optimize the local variational parameters obtained from an inference network when learning deep Boltzmann machines. For DLGMs, Hjelm et al. (2016) also consider the optimization of the local variational parameters, though their exposition focuses on deriving an importance-sampling-based bound to use during learning in deep generative models with discrete latent variables. Their experimental results suggest the procedure does not improve performance much on the binarized MNIST dataset. This is consistent with our experience—we found that our secondary optimization procedure helped more when modeling sparse, high-dimensional text data than when modeling MNIST.
Leveraging Gradient Information: The algorithmic procedure for obtaining Jacobian Vectors that we propose resembles that used to derive Fisher Score features. For an data point X under a parameteric distribution p(X; θ), the Fisher scores is defined as UX = ∇θ log p(X; θ). Jaakkola & Haussler (2007) similarly use UX to form a kernel function for subsequent use in a discriminative classifier. The intuition behind such methods is to note that the derivative of the log-probability with respect to the parameters of the generative model encodes all the variability in the input under the generative process. We rely on a related intuition, although our motivations are different; we are interested in characterizing isolated features such as words, not vector observations such as documents. Also, we consider Jacobians with respect to per-observation latent variables z, rather than globally shared parameters θ.
In the context of discriminative modeling, (Erhan et al., 2009) use gradient information to study the patterns with which neurons are activated in a deep neural networks while (Wang et al., 2016) use the spectra of the Jacobian to study the complexity of the functions learned by neural networks.
Introspection via Embeddings: Landauer et al. (1998) proposed latent semantic analysis, one of the earliest works to create vector space representations of documents. Bengio et al. (2003); Mikolov & Dean (2016) propose log-linear models to create word-representations from document corpora in an unsupervised fashion. Rudolph et al. (2016) describe a family of models to create contextual embeddings where the conditional distributions that lie in the exponential family. Finally, Choi et al. (2016) propose a variant of Word2Vec to create representations of diagnosis codes from temporal Electronic Health Record data. The models above explicitly condition the probability of a word on its nearby context. In contrast, our model models the probability of a word as it appears in the document (or rather, conditioned on its global context). Augmenting the generative model in Figure 1 to incorporate local context is a possible direction for future work.
Miao et al. (2016) learn a shallow log-linear model on text data and obtain embeddings for words from the weight matrix that parameterize their generative model. Li et al. (2016) propose a modification to LDA that explicitly models representations for word in addition to modeling the word-topic structure.
5 EVALUATION
Text Data: We study the effect of further optimization of the variational parameters and inference with tf-idf features on the two datasets of varying size: the smaller 20Newsgroups (Lang, 2008) (train/valid/test: 10768/500/7505, V : 2000) and the larger RCV2 (Lewis et al., 2004) dataset (train/valid/test: 789414/5000/10000, V : 10000). We follow the preprocessing procedure defined in (Miao et al., 2016) for both datasets. We also train models on the Wikipedia corpus used in (Huang et al., 2012). We remove stop words, words appearing less than ten times in the dataset. and select our vocabulary to comprise the union of the top 20000 words in the corpus, the words in the WordSim353
(Finkelstein et al., 2001) and the words in the Stanford Contextual Word Similarity Dataset (SCWS) (Huang et al., 2012). The resulting dataset is of size train/valid: 1212781/1000 and V :20253.
EHR Data: We train shallow and deep generative models on a dataset constructed from Electronic Medical Records. The dataset comprises 185000 patients where each patient’s data across time was aggregated to create a bag-of-diagnosis-codes representation of the patient. The vocabulary comprises four different kinds of medical diagnosis codes: ICD9 (diagnosis), LOINC (laboratory tests), NDC (prescription medication), CPT (procedures). For a single patient, we have 51321 diagnosis codes.
Training Procedure: On all datasets, we train shallow log-linear models (γ(z) = Wz + b) and deeper three-layer DLGMs (γ(z) = MLP(z; θ)). We vary the number of secondary optimization steps M = 1, 100 (cf. Algorithm 1) to study the effect of optimization on ψ(x) with ADAM (Kingma & Ba, 2015). We use a mini-batch size of 500, a learning rate of 0.01 for ψ(x) and 0.0008 for θ, φ. The inference network was fixed to a two-layer MLP whose intermediate hidden layer h(x) was used to parameterize the mean and diagonal log-variance µ(x), log Σ(x). To evaluate the quality of the learned generative models, we report an upper bound on perplexity (Mnih & Gregor, 2014) given by exp(− 1N ∑ i 1 Ni
log p(xi)) where log p(xi) is replaced by Eq 2. The notation 3-M100-tfidf indicates a model where the MLP parameterizing γ(z) has three hidden layers, the local variational parameters are updated 100 times before an update of θ and tf-idf features were used in the inference network.
Improving Learning: Table 1 depicts our results on 20newsgroups and RCV2. On the smaller dataset, we find that the deeper models overfit quickly and are outperformed by shallow generative models. On the larger datasets, the deeper models’ capacity is more readily utilized yielding better generalization. The use of tf-idf features always helps learning on smaller datasets. On larger datasets, the benefits are smaller when we also optimize ψ(x). Finally, the optimization of the local variational parameters appears to help most on the larger datasets. To investigate how this occurs, we plot the held-out likelihood versus epochs. For models trained on the larger RCV2 (Figure 2a) and Wikipedia (Figure 2b) datasets, the larger deep generative models converge to better solutions (and in fewer passes through the data) with the additional optimization of ψ(x).
To study the effect of investigate where optimizing ψ(x) is particularly effective on, we train a three layer model on different subsets of the Wikipedia dataset. The subsets are created by selecting the top K most frequently occurring features in the data. Our rationale is that by holding everything fixed and varying the level of sparsity (datasets with smaller values of K are less sparse) in the data, we can begin to understand when our method is most helpful. On held-out data, we compute the difference between the perplexity when the model is trained with M = 1 (denoted PM1) and M = 100 (denoted PM100) and compute the relative decrease in perplexity obtained as PM1−PM100PM100 . The results are depicted in Figure 2c where we see that our method improves learning as a function of the dimensionality of the data.
In Table 5 in the supplementary material, we study the effect of varying the parameters of the inference network. There, we perform a small grid search over the hidden dimension and the number of layers in the inference network and find that optimizing the variational parameters continues to produces models with lower overall perplexity.
Jacobian Vectors: Our first avenue for introspection into the learned generative model is using log-singular values of the Jacobian matrix. Since the Jacobian matrix precisely encodes how sensitive the outputs are with respect to the inputs, the log-singular value spectrum of this matrix directly captures the amount of variance in the data explained by the latent space. Said differently, we can read off the number of active units in the DLGM or VAE by counting the number of log-singular values larger than zero. Furthermore, this method of introspection depends only on the parameters of the generative model. In Figure 2d, 2e, we see that for larger models continuing to optimize the variational parameters allows us to learn models that use many more of the available latent dimensions. This suggests that, when fit to text data, DLGMs may be particularly susceptible to the overpruning phenomenon noted by Burda et al. (2015). In Figure 2, the lower held-out perplexity and the increased utilization of the latent space suggest that the continued optimization of the variational parameters yields more powerful generative models.
We investigate how the Jacobian matrix may be used for model introspection by studying the qualitative properties of J logmean on DLGMs (of type “3-M100-tfidf”) trained on two, diverse sets of data. We form a Monte Carlo estimate of J logmean using 400 samples. The cosine distance is
used to define neighbors of words in the embedding space of the Jacobian and spectral clustering (Von Luxburg, 2007) is used to form clusters.
In Table 2a, we visualize some of the nearest neighbors of words using J logmean obtained from models trained on the Wikipedia dataset. The neighbors are semantically sensible. Instead of evaluating the Jacobian at L points z1:L ∼ p(z), one may instead evaluate it at z1:L ∼ q(z|x) for some x. In Table 2b, we select three polysemous query words alongside “context words” that disambiguate the query’s meaning. For each word-context pair, we create a document comprising a subset of words in the the
(a) Word Embeddings (Nearest Neighbors): We visualize nearest neighbors of word embeddings. We exclude plurals of the query and other words in the neighborhood.
Query Neighborhood intelligence espionage, secrecy, interrogation, counterterrorism zen dharma, buddhism, buddhas, meditation,yoga artificial artificially, molecules, synthetic, soluble military civilian, armys, commanders, infantry
(b) Word Embeddings (Polysemy): We visualize the nearest neighbors under the Jacobian vector induced by the posterior distribution of a document created based on the context word.
Word Context Neighboring Words crane construction lifting, usaaf, spanned, crushed, lift bird erected, parkland, locally, farmland, causeway bank river watershed, footpath, confluence, drains, tributary money banking, government, bankers, comptroller, fiscal fires burn ignition, combustion, engines, fuel, engine
layoff thunderstorm, grassy, surrounded, walkway, burning (c) Medical Embeddings (Nearest Neighbors): We evaluate nearest neighbors of selected diagnosis, drug and procedure codes (ignoring duplicates and shortening some code names). Metformin, Glimepiride, Pioglitazone and Avandia are diabetic drugs. A contour meter is an instrument to track blood glucose. Advair, Albuterol, Proventil and Spiriva are prescribed to patients with chronic obstructive pulmonary disease (COPD)
context’s Wikipedia page. Then, we use the learned inference network to perform posterior inference to evaluate J logmean at the corresponding q(z|x). This yields a set of contextual Jacobian vectors. We display the nearest neighbors for each word under different contextual Jacobian vectors and find that, while not always perfect, they capture different contextually relevant semantics. The take-away here is that by combining posterior inference in this Bayesian network with our methodology of introspecting the model, one obtains different context-specific representations for the observations despite not having been trained to capture this explicitly.
In Table 8b (appendix), we visualize clusters formed from the embeddings of medical diagnosis codes to find that they exhibit topical coherence. In Table 2c, the nearest neighbors of drugs include other drugs prescribed in conjunction with or as a replacement for the query drug. For diagnosis codes such as “Asbestosis”, the nearest neighbors are symptoms and procedures associated with the
disease. Finally, for a qualitative evaluation of Jacobian vectors obtained from a model trained on movie ratings, we refer the reader to the appendix.
The Semantics of Embeddings: We evaluate the vector space representations that we obtain from Jmean on benchmarks (such as WordSim353 (Finkelstein et al., 2001) and SCWS (Huang et al., 2012)) that attempt to measure the similarity of words. The algorithmically derived measure of similarity is compared to a human-annotated score (between one and ten) using the Spearman rank correlation. The models that we compare to primarily use local context, which yields a more precise signal about the meanings of particular words. Closest to us in terms of training procedure is (Huang (G)) in Table 3a, whose model we outperform. Finding ways to incorporate local context is fertile ground for future work on models tailor-made for extracting embeddings.
For medical codes, we follow the method in (Choi et al., 2016). The authors build two kinds of evaluations to estimate whether an embedding space of medical diagnosis codes captures medically related concepts well. MRMNDF-RT (Medical Relatedness Measure under NDF-RT) leverages a database (NDF-RT) to evaluate how good an embedding space is at answering analogical queries between drugs and diseases such as uDiabetes ≈ uMetformin − (uLung Cancer − uTarceva). (Metformin is a diabetic drug and Tarceva is used in the treatment of lung cancer). The evaluation (MRMCCS) measures if the neighborhood of the diagnosis codes is medically coherent using a predefined medical ontology (CCS) as ground truth. The number computed may be thought of as a measure of precision, where a higher number is better. We refer the reader to the appendix for additional details.
Table 4 details the results on evaluating the medical embeddings. Once again, the baselines we compare (Choi et al., 2016) are variants of Word2Vec that maximize the likelihood of the diagnosis codes conditioned on carefully crafted contexts. Our method performs comparably to the baselines, even though it relies exclusively on global context and was not designed with this task in mind. This setting depicts an instance where Jacobian vectors resulting from a deeper, better-trained model outperform those from a shallow model, highlighting the importance of a method of interpretation agnostic to the structure of the conditional probability functions in the generative model.
Between the three choices of Jacobian vectors, we found that all three perform comparably on the word similarity evaluation with J probmean slightly outperforming the others. On the medical data, with we found similar results aside from a few cases where J probmean did not perform well. For deeper models, we found that optimizing ψ(x) improved the quality of the obtained Jacobian vectors on text and medical data. The full versions of Tables 3 and 4 can be found in the appendix.
6 DISCUSSION
We explored techniques to improve inference and learning in deep generative models of sparse non-negative data. We also developed and explored a novel, simple, yet effective method to interpret the structure of the non-linear generative model via embeddings obtained from the Jacobian matrix relating latent variables to observations. The embeddings are evaluated qualitatively and quantitatively, and were seen to exhibit interesting semantic structure across a variety of domains. Studying the effects of varying the priors on the latent variables, conditioning on context, and varying the neural architectures that parameterize the conditional distributions suggest avenues for blending ideas from generative modeling and Bayesian inference into building more powerful embeddings for data.
APPENDIX A RCV2: COMPLEXITY OF THE INFERENCE NETWORK
We study the effect of varying the inference network on the two-layer model trained with the RCV2 data. Folding fixed a three-layer DLGM with stochastic dimension 100 (same architecture as 3−tfidf in Table 1), we learn models on the RCV2 data using M = 1, 100 (we evaluate and report bounds on perplexity using M = 100) and display the results in Table 5. When M = 1 (the standard procedure for training VAEs and DLGMs), increasing the number of layers in the inference network decreases the quality of the model learned. One possible explanation for this is that the already noisy gradients of the inference network must propagate along a longer path in a deeper inference network, slowing down learning of the parameters φ which in turn affects the quality of inference. In contrast, increasing hidden dimension of the inference network improves results. Generally, we obtain better results (in both train and validation error) with M = 100 than training with M = 1 across the various configurations of the inference network that we tried. Furthermore, we find that when M = 100, the inference network architecture is less relevant and all models converge to approximately the same result suggesting that the procedure treats the output of the inference network as a crude initialization for the variational parameters and that the subsequent steps of optimization are primarily responsible for gains in learning.
APPENDIX B NETFLIX: EMBEDDINGS FOR MOVIES
The Netflix dataset Netflix (2009) comprises movie ratings of 500, 000 users. We treat each user’s ratings as a document and model the numbers ascribed to each movie (from 1− 5) as counts drawn from the multinomial distribution parameterized as in Eq. 1. We train a three-layer DLGM on the dataset, evaluate Jmean with 100 samples and consider two distinct methods of evaluating the learned embeddings. We cluster the movie embeddings (using spectral clustering with cosine distance to obtain 100 clusters) and depict some of the clusters in Table 6a. We find that clusters exhibit coherent themes such as documentary films, horror and James Bond movies. Other clusters (not displayed) included multiple seasons of the same show such as Friends, WWE wrestling, and Pokemon. In Table 6b, we visualize the neighbors of some popular films. In the examples we visualize, the nearest neighbors include sequels, movies from the same franchise or, as in the case of 12 Angry Men, other dramatic classics.
To compare the effect of using a model to create embeddings versus using the raw data from a large dataset directly, we evaluated nearest neighbors of movies using a simple baseline. For a query movie, we found all users who gave the movie a rating of 3 or above (nominally, they watched and liked the movie). Then, for all those users, we computed the mean ratings they gave to every other movie in the vocabulary and ranked them based on the mean ratings. We display the top five movies obtained using this approach in Table 6. The query words are the same as in Table 6b. For most of the queries, the difference between the two is evident and we simply end up with popular, well-liked movies rather than relevant movies.
APPENDIX C WIKIPEDIA: EMBEDDINGS FOR WORD
Table 7 is the full version of Table 3 in the main paper. We find that the three variants of the Jacobian vectors perform comparably across the board. The vectors obtained from shallow log-linear models appear to have the edge. The evaluation on the WordSim and SCWS datasets are done by computing the Spearman rank correlation between human annotated rankings between 1 and 10 and an algorithmically derived measures of word-pair similarity. We first compute the distances between all word pairs. Our measure of similarity is obtained by subtracting the distances from the maximal distance across all word pairs.
APPENDIX D EHR DATA: EMBEDDINGS FOR DIAGNOSIS CODES
For EHR data in particular, the bag-of-diagnosis-codes assumption we make is a crude one since (1) we assume the temporal nature of the patient data is irrelevant, and (2) combining patient statistics over time renders it difficult for the generative model to disambiguate the correlations between codes that correspond to multiple diseases a patient may suffer from. Despite this, it is interesting that the Jacobian vectors still capture much of the meaningful structure among the diagnosis codes (c.f Table 2c, 8b). Here we provide additional details surrounding the evaluating of medical embeddings.
MRMCCS(V,G): The Agency for Healthcare Research and Quality’s clinical classification software (CCS) collapses the hierarchical ICD9 diagnosis codes into clinically meaningful categories. The evaluation on CCS checks whether the nearest neighbors of a disease include other diseases related to it (if they are in the same category in the CCS). Using the ICD9 hierarchy, the authors further split the evaluation task into predicting neighbors of fine-grained and coarse grained diagnosis codes.
For a choice of granularity G ∈ {fine,coarse}, V (G) ∈ V denotes the subset of ICD9 codes in the vocabulary. IG(v(i)) is one if the v’s i’th nearest neighbor: v(i) is in the same group as v according to G.
MRMCCS(V,G) = 1 |V (G)| ∑
v∈V (G) 40∑ k=1 IG(v(i)) log2(i+ 1)
(6)
MRMNDF-RT(V,R): The other evaluation uses the National Drug File Reference Terminology (NDF-RT) to evaluate analogical reasoning. The NDF-RT provides two kinds of relationships (R) between drugs and diseases: May-Treat (if the drug may be used to treat the disease) and May-Prevent. Given φA as the embedding for a code A, this test automates the evaluation of analogies such as φDiabetes︸ ︷︷ ︸
r
≈ φMetformin︸ ︷︷ ︸ v −(φLung Cancer − φTarceva︸ ︷︷ ︸ s ). Here v is the query code and s is a representation
of the relationship we seek. (Metformin is a diabetic drug and Tarceva is used in the treatment of lung cancer.) The evaluation we perform reports a number proportional to the number of times
the neighborhood of v − s contains r for the best value of s (computed from the set of all valid drug-disease relationships in the datasets.) Given V ∗ ∈ V (concepts for which NDF-RT has at-least one substance with the given relation), IR ( ∪40i=1(v − s)(i) ) is one if any of the medical concepts in the top-40 neighborhood of the selected medical concept v satisfies relation R.
MRMNDF-RT(V,R) = 1 |V ∗| ∑ v∈V ∗ IR ( ∪40i=1(v − s)(i) ) (7)
In both cases the choice of 40 was adopted to maintain consistency with (Choi et al., 2016). The evaluation is conducted by taking the average result over all possible seeds s and the best possible seed s for a query.
(a) Medical Analogies: We can perform analogical reasoning with embeddings of medical codes. If we know a drug used to treat a disease, we can use their relationship in vector space to find unknown drugs associated with a different disease. The queries are of the form Code 1→Code 2 =⇒ Code 3→?. Sicca syndrome or Sjogren’s disease is an immune disease treated with Evoxac and Methotrexate is commonly used to treat Rheumatoid Arthiritis. “ Leg Varicosity” denotes the presence of swollen veins under the skin. “Ligation of angioaccess arteriovenous fistula” denotes the tying of passage between an artery and a vein.
(b) Medical Embeddings (Clustering): We visualize some topical clusters of diagnosis codes.
Label Diagnosis Codes Thrombosis Hx Venous Thrombosis, Compression Of Vein, Renal Vein Thrombosis Occular Atrophy Optic Atrophy, Retina Layer Separation, Chronic Endophthalmitis Drug Use Opioid Dependence, Alcohol Abuse-Continuous, Hallucinogen Dep

Review:
The paper claims improved inference for density estimation of sparse data (here text documents) using deep generative Gaussian models (variational auto-encoders), and a method for deriving word embeddings from the model's generative parameters that allows for a degree of interpretability similar to that of Bayesian generative topic models.
To discuss the contributions I will quickly review the generative story in the paper: first a K-dimensional latent representation is sampled from a multivariate Gaussian, then an MLP (with parameters \theta) predicts unnormalised potentials over a vocabulary of V words, the potentials are exponentiated and normalised to make the parameters of a multinomial from where word observations are repeatedly sampled to make a document. Here intractable inference is replaced by the VAE formulation where an inference network (with parameters \phi) independently predicts for each document the mean and variance of a normal distribution (amenable to reparameterised gradient computation).
The first, and rather trivial, contribution is to use tf-idf features to inject first order statistics (a global information) into local observations. The authors claim that this is particularly helpful in the case of sparse data such as text.
The second contribution is more interesting. In optimising generative parameters (\theta) and variational parameters (\phi), the authors turn to a treatment which is reminiscent of the original SVI procedure. That is, they see the variational parameters \phi as *global* variational parameters, and the predicted mean \mu(x) and covariance \Sigma(x) of each observation x are treated as *local* variational parameters. In the original VAE, local parameters are not directly optimised, instead they are indirectly optimised via optimisation of the global parameters utilised in their prediction (shared MLP parameters). Here, local parameters are optimised holding generative parameters fixed (line 3 of Algorithm 1). The optimised local parameters are then used in the gradient step of the generative parameters (line 4 of Algorithm 1). Finally, global variational parameters are also updated (line 5). 
Whereas indeed other authors have proposed to optimise local parameters, I think that deriving this procedure from the more familiar SVI makes the contribution less of a trick and easier to relate to.
Some things aren't entirely clear to me. I think it would have been nice if the authors had shown the functional form of the gradient used in step 3 of Algorithm 1. The gradient step for global variational parameters (line 5 of Algorithm 1) uses the very first prediction of local parameters (thus ignoring the optimisation in step 3), this is unclear to me. Perhaps I am missing a fundamental reason why that has to be the case (either way, please clarify).
The authors argue that this optimisation turns out helpful to modelling sparse data because there is evidence that the generative model p_\theta(x|z) suffers from poor initialisation. Please, discuss why you expect the initialisation problem to be worse in the case of sparse data.
The final contribution is a neat procedure to derive word embeddings from the generative model parameters. These embeddings are then used to interpret what the model has learnt. Interestingly, these word embeddings are context-sensitive once that the latent variable models an entire document.
About Figures 2a and 2b: the caption says that solid lines indicate validation perplexity for M=1 (no optimisation of local parameters) and dashed lines indicate M=100 (100 iterations of optimisation of local parameters), but the legends of the figures suggest a different reading. If I interpret the figures based on the caption, then it seems that indeed deeper networks exposed to more data benefit from optimisation of local parameters. Are the authors pretty sure that in Figure 2b models with M=1 have reached a plateau (so that longer training would not allow them to catch up with M=100 curves)? As the authors explain in the caption, x-axis is not comparable on running time, thus the question.
The analysis of singular values seems like an interesting way to investigate how the model is using its capacity.  However, I can barely interpret Figures 2c and 2d, I think the authors could have walked readers through them.
As for the word embedding I am missing an evaluation on a predictive task. Also, while illustrative, Table 2b is barely reproducible. The text reads "we create a document comprising a subset of words in the the context’s Wikipedia page." which is rather vague. I wonder whether this construct needs to be carefully designed in order to get Table 2b.
In sum, I have a feeling that the inference technique and the embedding technique are both useful, but perhaps they should have been presented separately so that each could have been explored in greater depth.