Original: K ERNEL I MPLICIT V ARIATIONAL I NFERENCE
Revised: K ERNEL I MPLICIT V ARIATIONAL I NFERENCE

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Recent progress in variational inference has paid much attention to the ﬂexibility of variational posteriors.
Revised: Recent progress in variational inference has paid much attention to the ﬂexibility of variational posteriors.

Original: One promising direction is to use implicit distributions, i.e., distributions without tractable densities as the variational posterior.
Revised: One promising direction is to use implicit distributions, i.e., distributions without tractable densities as the variational posterior.

Original: However, existing methods on implicit posteriors still face challenges of noisy estimation and can hardly scale to high-dimensional latent variable models.
Revised: However, existing methods on implicit posteriors still face challenges of noisy estimation and can hardly scale to high-dimensional latent variable models.

Original: In this paper, we present an new approach named Kernel Implicit Variational Inference that addresses these challenges.
Revised: In this paper, we present an new approach named Kernel Implicit Variational Inference that addresses these challenges.

Original: As far as we know, for the ﬁrst time implicit variational inference is successfully applied to Bayesian neural networks, which shows promising results on both regression and classiﬁcation tasks.
Revised: As far as we know, for the ﬁrst time implicit variational inference is successfully applied to Bayesian neural networks, which shows promising results on both regression and classiﬁcation tasks.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Bayesian methods have been playing vital roles in machine learning by providing a principled approach for uncertainty modeling, posterior inference and preventing over-ﬁtting (Ghahramani, 2015).
Revised: Bayesian methods have been playing vital roles in machine learning by providing a principled approach for uncertainty modeling, posterior inference and preventing over-ﬁtting (Ghahramani, 2015).

Original: As it becomes a common practice to build deep models that have many parameters (LeCun et al., 2015), it is even more important to have a Bayesian formulation to protect these models.
Revised: As it becomes a common practice to build deep models that have many parameters (LeCun et al., 2015), it is even more important to have a Bayesian formulation to protect these models.

Original: For example, Bayesian Neural Networks (BNNs) (Neal, 2012; Blundell et al., 2015) have shown promise in dealing with model uncertainty and learning with few labeled data.
Revised: For example, Bayesian Neural Networks (BNNs) (Neal, 2012; Blundell et al., 2015) have shown promise in dealing with model uncertainty and learning with few labeled data.

Original: Another recent trend is to incorporate deep neural networks as a powerful function mapping between random variables in a
Revised: Another recent trend is to incorporate deep neural networks as a powerful function mapping between random variables in a

Original: Bayesian network, such as deep generative models like variational autoencoders (VAE) (Kingma &
Revised: Bayesian network, such as deep generative models like variational autoencoders (VAE) (Kingma &

Original: Welling, 2013).
Revised: Welling, 2013).

Original: Except a few simple examples, Bayesian inference is typically challenging, for which variational inference (VI) has been a standard workhorse to approximate the true posterior (Zhu et al., 2017).
Revised: Except a few simple examples, Bayesian inference is typically challenging, for which variational inference (VI) has been a standard workhorse to approximate the true posterior (Zhu et al., 2017).

Original: Traditional VI focuses on factorized variational posteriors to get analytical updates.
Revised: Traditional VI focuses on factorized variational posteriors to get analytical updates.

Original: While recent progress in this ﬁeld drives VI into stochastic, differentiable and amortized (Hoffman et al., 2013;
Revised: While recent progress in this ﬁeld drives VI into stochastic, differentiable and amortized (Hoffman et al., 2013;

Original: Paisley et al., 2012; Mnih & Gregor, 2014; Kingma & Welling, 2013), which does not rely on analytical updates anymore, factorized posteriors are still commonly used as the variational family.
Revised: Paisley et al., 2012; Mnih & Gregor, 2014; Kingma & Welling, 2013), which does not rely on analytical updates anymore, factorized posteriors are still commonly used as the variational family.

Original: This greatly restricts the ﬂexibility of the variational posterior, especially in high-dimensional spaces, which often leads to biased inference.
Revised: This greatly restricts the ﬂexibility of the variational posterior, especially in high-dimensional spaces, which often leads to biased inference.

Original: There has been some works that try to improve the ﬂexibility of variational posteriors, borrowing ideas from invertible transformation of probability distributions (Rezende & Mohamed, 2015; Kingma et al., 2016; Tomczak & Welling, 2016).
Revised: There has been some works that try to improve the ﬂexibility of variational posteriors, borrowing ideas from invertible transformation of probability distributions (Rezende & Mohamed, 2015; Kingma et al., 2016; Tomczak & Welling, 2016).

Original: In their works, it is important for the transformation to be invertible to ensure that the transformed distribution has a tractable density and can be used as the variational posterior.
Revised: In their works, it is important for the transformation to be invertible to ensure that the transformed distribution has a tractable density and can be used as the variational posterior.

Original: Although utilizing invertible transformation is a promising direction to increase the expressiveness of the variational posterior, we argue that a more ﬂexible variational family can be constructed by using general deterministic or stochastic transformations, which is not necessarily invertible.
Revised: Although utilizing invertible transformation is a promising direction to increase the expressiveness of the variational posterior, we argue that a more ﬂexible variational family can be constructed by using general deterministic or stochastic transformations, which is not necessarily invertible.

Original: As a common result, the variational posterior we get in this way does not admit a tractable likelihood, despite there is a way to sample from it.
Revised: As a common result, the variational posterior we get in this way does not admit a tractable likelihood, despite there is a way to sample from it.

Original: This kind of distribution is called implicit distributions, and for variational methods that use an implicit variational posterior (also known as variational programs (Ranganath et al., 2016), wild variational approximations (Liu & Feng, 2016)), we refer to them as
Revised: This kind of distribution is called implicit distributions, and for variational methods that use an implicit variational posterior (also known as variational programs (Ranganath et al., 2016), wild variational approximations (Liu & Feng, 2016)), we refer to them as

Original: Implicit Variational Inference (implicit VI).
Revised: Implicit Variational Inference (implicit VI).

Original: In this paper we present an approach named Kernel Implicit Variational Inference (KIVI), which addresses the noisy estimation and scalability problems of implicit VI.
Revised: In this paper we present an approach named Kernel Implicit Variational Inference (KIVI), which addresses the noisy estimation and scalability problems of implicit VI.

Original: KIVI is applicable to both global and local latent variable models, which is demonstrated by experiments on BNNs and VAEs.
Revised: KIVI is applicable to both global and local latent variable models, which is demonstrated by experiments on BNNs and VAEs.

Original: As far as we know, this is the ﬁrst time that implicit VI is successfully applied to BNNs, which shows promising results on both regression and classiﬁcation tasks.
Revised: As far as we know, this is the ﬁrst time that implicit VI is successfully applied to BNNs, which shows promising results on both regression and classiﬁcation tasks.

Original:  B ACKGROUND
Revised:  B ACKGROUND

Original: Consider a generative model p ( z , x )
Revised: Consider a generative model p ( z , x )

Original: = p ( z ) p ( x | z ) , where x and z denotes observed and latent variables, respectively.
Revised: = p ( z ) p ( x | z ) , where x and z denotes observed and latent variables, respectively.

Original: In VI, a variational distribution q φ ( z ) in some parametric family is chosen to approximate the true posterior p ( z | x ) by optimizing the evidence lower bound (ELBO):
Revised: In VI, a variational distribution q φ ( z ) in some parametric family is chosen to approximate the true posterior p ( z | x ) by optimizing the evidence lower bound (ELBO):

Original: [Equation]
Revised: [Equation]

Original: where KL denotes the Kullback-Leibler divergence KL( q (cid:107) p )
Revised: where KL denotes the Kullback-Leibler divergence KL( q (cid:107) p )

Original: =
Revised: =

Original: E q (cid:104) log qp (cid:105) .
Revised: E q (cid:104) log qp (cid:105) .

Original: This objective is a lower bound of the log-likelihood log p ( x ) since it can be written as L ( x ; φ )
Revised: This objective is a lower bound of the log-likelihood log p ( x ) since it can be written as L ( x ; φ )

Original: = log p ( x ) −
Revised: = log p ( x ) −

Original: KL( q φ ( z ) (cid:107) p ( z | x )) .
Revised: KL( q φ ( z ) (cid:107) p ( z | x )) .

Original: The maximum of this objective is achieved when q φ ( z ) =
Revised: The maximum of this objective is achieved when q φ ( z ) =

Original: p ( z | x ) .
Revised: p ( z | x ) .

Original: From Eq.
Revised: From Eq.

Original: (1), we can see that the challenge of using an implicit q φ is that computing KL( q φ ( z ) (cid:107) p ( z )) requires to evaluate the density of q φ , which is intractable for an implicit distribution.
Revised: (1), we can see that the challenge of using an implicit q φ is that computing KL( q φ ( z ) (cid:107) p ( z )) requires to evaluate the density of q φ , which is intractable for an implicit distribution.

Original: Recently, inspired by the probabilistic interpretation of Generative Adversarial Networks (GAN) (Goodfellow et al., 2014; Mohamed & Lakshminarayanan, 2016), there has been some works that extend the GAN approach to the posterior inference of latent variable models (LVMs) (Mescheder et al., 2017; Husz´ar, 2017; Tran et al., 2017).
Revised: Recently, inspired by the probabilistic interpretation of Generative Adversarial Networks (GAN) (Goodfellow et al., 2014; Mohamed & Lakshminarayanan, 2016), there has been some works that extend the GAN approach to the posterior inference of latent variable models (LVMs) (Mescheder et al., 2017; Husz´ar, 2017; Tran et al., 2017).

Original: These methods all use an implicit variational family and thus can be categorized into implicit VI methods.
Revised: These methods all use an implicit variational family and thus can be categorized into implicit VI methods.

Original: One of their key observations is that the density ratio q φ ( z ) p ( z ) can be estimated from samples of the two distributions by using a probabilistic classiﬁer.
Revised: One of their key observations is that the density ratio q φ ( z ) p ( z ) can be estimated from samples of the two distributions by using a probabilistic classiﬁer.

Original: They ﬁrst assign class labels ( y ) to q and p : Let samples from q φ ( z ) be of class y = 1 , and samples from p ( z ) be of class y = 0 .
Revised: They ﬁrst assign class labels ( y ) to q and p : Let samples from q φ ( z ) be of class y = 1 , and samples from p ( z ) be of class y = 0 .

Original: Given equal class priors, the density ratio at a given point can be calculated as q φ ( z ) /p
Revised: Given equal class priors, the density ratio at a given point can be calculated as q φ ( z ) /p

Original: ( z ) =
Revised: ( z ) =

Original: p ( z | y = 1) /p
Revised: p ( z | y = 1) /p

Original: ( z | y = 0) =
Revised: ( z | y = 0) =

Original: p ( y = 1 | z ) /p
Revised: p ( y = 1 | z ) /p

Original: ( y = 0 | z ) , which is the ratio between the class probabilities given the data point.
Revised: ( y = 0 | z ) , which is the ratio between the class probabilities given the data point.

Original: To estimate this, a probabilistic classiﬁer D is trained to classify between the two classes, with a logistic loss:
Revised: To estimate this, a probabilistic classiﬁer D is trained to classify between the two classes, with a logistic loss:

Original: [Equation]
Revised: [Equation]

Original: where D ( z ) is a classiﬁer that outputs the probability of z ’s being from class y = 1 .
Revised: where D ( z ) is a classiﬁer that outputs the probability of z ’s being from class y = 1 .

Original: Given
Revised: Given

Original: D is ﬂexible enough, the optimal solution of problem (2) is D ( z ) =
Revised: D is ﬂexible enough, the optimal solution of problem (2) is D ( z ) =

Original: q φ ( z ) / ( q φ ( z ) + p ( z )) .
Revised: q φ ( z ) / ( q φ ( z ) + p ( z )) .

Original: Therefore, the KL divergence term in the ELBO of Eq.
Revised: Therefore, the KL divergence term in the ELBO of Eq.

Original: (1) can be approximated as KL( q φ (cid:107) p )
Revised: (1) can be approximated as KL( q φ (cid:107) p )

Original: ≈ E q φ ( z ) [log D ( z ) − log(1 − D ( z ))] .
Revised: ≈ E q φ ( z ) [log D ( z ) − log(1 − D ( z ))] .

Original: This is called prior-contrastive forms of VI in Husz ´ ar (2017).
Revised: This is called prior-contrastive forms of VI in Husz ´ ar (2017).

Original: Note that the ratio approximation doesn’t change the gradients once the approximation is accurate (Husz ´ ar, 2017).
Revised: Note that the ratio approximation doesn’t change the gradients once the approximation is accurate (Husz ´ ar, 2017).

Original: Though incorporating the discriminative power in a probabilistic model has shown great success in GANs, this method still suffers from challenging problems when applied to VI:
Revised: Though incorporating the discriminative power in a probabilistic model has shown great success in GANs, this method still suffers from challenging problems when applied to VI:

Original: • Noisy density ratio estimation (DRE)
Revised: • Noisy density ratio estimation (DRE)

Original: In VI, the variational posterior gets updated in each iteration.
Revised: In VI, the variational posterior gets updated in each iteration.

Original: To produce accurate estimation, the classiﬁer should be trained to optimum after each iteration.
Revised: To produce accurate estimation, the classiﬁer should be trained to optimum after each iteration.

Original: However, in practice the classiﬁer is only trained for one or several iterations for each variational update.
Revised: However, in practice the classiﬁer is only trained for one or several iterations for each variational update.

Original: It is hard for the classiﬁer to catch up with the variational posterior.
Revised: It is hard for the classiﬁer to catch up with the variational posterior.

Original: This unstable loss often produces noisy gradients and leads to unsatisfying results.
Revised: This unstable loss often produces noisy gradients and leads to unsatisfying results.

Original: Besides, even if the classiﬁer quickly achieves the optimum in a small number of iterations, there is still another issue.
Revised: Besides, even if the classiﬁer quickly achieves the optimum in a small number of iterations, there is still another issue.

Original: Notice that the training loss in problem (2) is with expectations.
Revised: Notice that the training loss in problem (2) is with expectations.

Original: But in practice we are using samples from the two distributions to approximate it.
Revised: But in practice we are using samples from the two distributions to approximate it.

Original: When the support of the distributions is high-dimensional, given the limited number of samples we use, the variance of this estimate is considerable, i.e., the classiﬁer may overﬁt the samples.
Revised: When the support of the distributions is high-dimensional, given the limited number of samples we use, the variance of this estimate is considerable, i.e., the classiﬁer may overﬁt the samples.

Original: The phenomenon is that the classiﬁer achieves a state where samples are easily distinguished and the probabilities given by the classiﬁer are near 0 or 1, which is commonly observed in experiments (Mescheder et al., 2017).
Revised: The phenomenon is that the classiﬁer achieves a state where samples are easily distinguished and the probabilities given by the classiﬁer are near 0 or 1, which is commonly observed in experiments (Mescheder et al., 2017).

Original: • High dimensional latent variables
Revised: • High dimensional latent variables

Original: As the density ratio is estimated by a classiﬁer, the samples from the two distributions of latent variables should be fed into it.
Revised: As the density ratio is estimated by a classiﬁer, the samples from the two distributions of latent variables should be fed into it.

Original: However, the typically used neural network classiﬁer cannot scale towards very high-dimensional inputs (e
Revised: However, the typically used neural network classiﬁer cannot scale towards very high-dimensional inputs (e

Original: g., parameters in a moderate-size Bayesian neural networks).
Revised: g., parameters in a moderate-size Bayesian neural networks).

Original:  K ERNEL I MPLICIT V ARIATIONAL I NFERENCE
Revised:  K ERNEL I MPLICIT V ARIATIONAL I NFERENCE

Original: To address the above challenges for Implicit VI, we propose to replace the probabilistic classiﬁer with a kernel method for DRE.
Revised: To address the above challenges for Implicit VI, we propose to replace the probabilistic classiﬁer with a kernel method for DRE.

Original: When this approach is applied to Bayesian neural networks, we further design a highly correlated implicit variational family that can scale to very large networks.
Revised: When this approach is applied to Bayesian neural networks, we further design a highly correlated implicit variational family that can scale to very large networks.

Original: Speciﬁcally, the key problem of the noisy estimation is partly due to the crude truncation of the inner loop after each variational update.
Revised: Speciﬁcally, the key problem of the noisy estimation is partly due to the crude truncation of the inner loop after each variational update.

Original: Then if a method for estimating the KL term can give closed-form solutions while maintaining reasonable accuracy, it should enable an algorithm without inner loops, which is more stable and potentially leads to better performance.
Revised: Then if a method for estimating the KL term can give closed-form solutions while maintaining reasonable accuracy, it should enable an algorithm without inner loops, which is more stable and potentially leads to better performance.

Original: Based on the intuition, we apply a kernel regression approach to estimate the density ratio function.
Revised: Based on the intuition, we apply a kernel regression approach to estimate the density ratio function.

Original: It has a closed-form solution that is guaranteed to be globally optimal.
Revised: It has a closed-form solution that is guaranteed to be globally optimal.

Original: 3. E STIMATING THE KL T ERM
Revised: 3. E STIMATING THE KL T ERM

Original: Speciﬁcally, let z ∈ R d be the latent variable, and the true density ratio is r ( z )
Revised: Speciﬁcally, let z ∈ R d be the latent variable, and the true density ratio is r ( z )

Original: =
Revised: =

Original: q ( z ) /p
Revised: q ( z ) /p

Original: ( z ) .
Revised: ( z ) .

Original: Consider modeling it with a function ˆ r ∈ H , where H is a Reproducing Kernel Hilbert Space (RKHS) induced by a positive deﬁnite kernel k ( z , z (cid:48) ) :
Revised: Consider modeling it with a function ˆ r ∈ H , where H is a Reproducing Kernel Hilbert Space (RKHS) induced by a positive deﬁnite kernel k ( z , z (cid:48) ) :

Original: R d × R d → R .
Revised: R d × R d → R .

Original: Similar to kernel ridge regression, we use an objective composed of a square loss for regression plus a penalty for the complexity of the function, where the complexity is measured by the RKHS norm ( (cid:107) ˆ r (cid:107)
Revised: Similar to kernel ridge regression, we use an objective composed of a square loss for regression plus a penalty for the complexity of the function, where the complexity is measured by the RKHS norm ( (cid:107) ˆ r (cid:107)

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Here λ is the regularization coefﬁcient.
Revised: Here λ is the regularization coefﬁcient.

Original: For the squared loss we choose the form used by the unconstrained Least Square Importance Fitting (uLSIF) (Kanamori et al., 2009):
Revised: For the squared loss we choose the form used by the unconstrained Least Square Importance Fitting (uLSIF) (Kanamori et al., 2009):

Original: [Equation]
Revised: [Equation]

Original: (4) where C is a constant.
Revised: (4) where C is a constant.

Original: Then L (ˆ r ) is deﬁned by the Monte Carlo estimate of J (ˆ r ) , using samples from p and q : L (ˆ r ) =
Revised: Then L (ˆ r ) is deﬁned by the Monte Carlo estimate of J (ˆ r ) , using samples from p and q : L (ˆ r ) =

Original: ˆ J (ˆ r ) = 12 n p (cid:80) n p
Revised: ˆ J (ˆ r ) = 12 n p (cid:80) n p

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ( z
Revised: ( z

Original: [Equation]
Revised: [Equation]

Original: ∼ q ( z ) .
Revised: ∼ q ( z ) .

Original: Proposition 1.
Revised: Proposition 1.

Original: The optimal solution of Eq.
Revised: The optimal solution of Eq.

Original: (3) lies in the linear subspace spanned by the kernel functions with the samples ( { z pi } n p i =1 , { z qj } n q j =1 ) as bases, i.e., ˆ r has the form:
Revised: (3) lies in the linear subspace spanned by the kernel functions with the samples ( { z pi } n p i =1 , { z qj } n q j =1 ) as bases, i.e., ˆ r has the form:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: This can be seen as the generalization of the representer theorem (Sch ¨ olkopf et al., 2001) to the density ratio problem.
Revised: This can be seen as the generalization of the representer theorem (Sch ¨ olkopf et al., 2001) to the density ratio problem.

Original: So the proof follows the same procedure.
Revised: So the proof follows the same procedure.

Original: See Appendix A.
Revised: See Appendix A.

Original: Substitute Eq.
Revised: Substitute Eq.

Original: (5) into Eq.
Revised: (5) into Eq.

Original: (3) and let the derivatives w.r.t.
Revised: (3) and let the derivatives w.r.t.

Original: α and β be zeros, we get the optimal solution (a detailed derivation is given in Appendix B):
Revised: α and β be zeros, we get the optimal solution (a detailed derivation is given in Appendix B):

Original: [Equation]
Revised: [Equation]

Original: where ( K p ) i,j = k
Revised: where ( K p ) i,j = k

Original: ( z pi , z pj ) , ( K pq ) i,j = k
Revised: ( z pi , z pj ) , ( K pq ) i,j = k

Original: ( z pi , z qj ) , and ( K q ) i,j
Revised: ( z pi , z qj ) , and ( K q ) i,j

Original: = k
Revised: = k

Original: ( z qi , z qj ) .
Revised: ( z qi , z qj ) .

Original: We use RBF kernels k
Revised: We use RBF kernels k

Original: ( z , z (cid:48) ) =
Revised: ( z , z (cid:48) ) =

Original: exp (cid:0) −(cid:107)
Revised: exp (cid:0) −(cid:107)

Original: z − z (cid:48) (cid:107) 22 / 2 σ 2 (cid:1) .
Revised: z − z (cid:48) (cid:107) 22 / 2 σ 2 (cid:1) .

Original: σ is the kernel bandwidth, which is determined by the commonly used median heuristic (the median of pairwise distances of the data points).
Revised: σ is the kernel bandwidth, which is determined by the commonly used median heuristic (the median of pairwise distances of the data points).

Original: Now that we have the approximate density ratio function ˆ r , a Monte Carlo estimate of the KL divergence term can be constructed by 1 n q (cid:80) n q i =1 log ˆ r
Revised: Now that we have the approximate density ratio function ˆ r , a Monte Carlo estimate of the KL divergence term can be constructed by 1 n q (cid:80) n q i =1 log ˆ r

Original: ( z qi ) .
Revised: ( z qi ) .

Original: Note that there is a constraint that the estimated density ratio should be non-negative.
Revised: Note that there is a constraint that the estimated density ratio should be non-negative.

Original: However, we do not involve it in the optimization objective in order to get a closed-form solution, which indicates that some post-processing should be done to ensure this property.
Revised: However, we do not involve it in the optimization objective in order to get a closed-form solution, which indicates that some post-processing should be done to ensure this property.

Original: We solve this issue by clipping the estimated density ratio.
Revised: We solve this issue by clipping the estimated density ratio.

Original: The clipping values are searched from { 10 − 8 , 10 − 16 , 10 − 32 } .
Revised: The clipping values are searched from { 10 − 8 , 10 − 16 , 10 − 32 } .

Original: In experiments we found that the algorithm is not sensitive to the clipping value.
Revised: In experiments we found that the algorithm is not sensitive to the clipping value.

Original: This is due to the accurate estimation guaranteed by the global optimal in the RKHS, which is a universal family when RBF kernels are used (Carmeli et al., 2010).
Revised: This is due to the accurate estimation guaranteed by the global optimal in the RKHS, which is a universal family when RBF kernels are used (Carmeli et al., 2010).

Original: The reverse ratio trick Another technique is essential to get an accurate estimate of the KL divergence, which we call the reverse ratio trick.
Revised: The reverse ratio trick Another technique is essential to get an accurate estimate of the KL divergence, which we call the reverse ratio trick.

Original: The key observation is that when the above method is applied, the optimization objective in Eq.
Revised: The key observation is that when the above method is applied, the optimization objective in Eq.

Original: (3) puts more weights into regions where the probability mass of p is high.
Revised: (3) puts more weights into regions where the probability mass of p is high.

Original: However, the KL term KL( q (cid:107) p ) to estimate chooses q as the base measure.
Revised: However, the KL term KL( q (cid:107) p ) to estimate chooses q as the base measure.

Original: Unless p and q match very well in where they put most probabilities, the ratio estimation objective does not ﬁt well with the KL-divergence targets.
Revised: Unless p and q match very well in where they put most probabilities, the ratio estimation objective does not ﬁt well with the KL-divergence targets.

Original: The solution is by a simple trick.
Revised: The solution is by a simple trick.

Original: Instead of estimating qp , we choose to estimate p q and compute the KL term as − E q log p q .
Revised: Instead of estimating qp , we choose to estimate p q and compute the KL term as − E q log p q .

Original: We denote the estimated reverse density ratio as ˆ r pq , then the corresponding KL estimate is − E q log ˆ r pq .
Revised: We denote the estimated reverse density ratio as ˆ r pq , then the corresponding KL estimate is − E q log ˆ r pq .

Original: We ﬁnd in experiments that this trick is very essential to make the estimation accurate enough for VI.
Revised: We ﬁnd in experiments that this trick is very essential to make the estimation accurate enough for VI.

Original: Gradient computation We now consider how to estimate the gradient of the KL term w.r.t.
Revised: Gradient computation We now consider how to estimate the gradient of the KL term w.r.t.

Original: variational parameters φ .
Revised: variational parameters φ .

Original: First it’s easy to prove as in Husz´ar (2017) that
Revised: First it’s easy to prove as in Husz´ar (2017) that

Original: [Equation]
Revised: [Equation]

Original: We also include it in Appendix C. Eq.
Revised: We also include it in Appendix C. Eq.

Original: ( 7) indicates that we can use any approximation of the density ratio, and the gradients w.r.t.
Revised: ( 7) indicates that we can use any approximation of the density ratio, and the gradients w.r.t.

Original: φ won’t change as long as the approximation is accurate.
Revised: φ won’t change as long as the approximation is accurate.

Original: We now replace the density ratio on the right side with ˆ r pq : ∇ φ KL( q φ (cid:107) p )
Revised: We now replace the density ratio on the right side with ˆ r pq : ∇ φ KL( q φ (cid:107) p )

Original: ≈ −∇ φ E q φ log ˆ r pq .
Revised: ≈ −∇ φ E q φ log ˆ r pq .

Original: Then, the reparameterization trick (Kingma & Welling, 2013) can be used:
Revised: Then, the reparameterization trick (Kingma & Welling, 2013) can be used:

Original: [Equation]
Revised: [Equation]

Original: 3. T HE A LGORITHM
Revised: 3. T HE A LGORITHM

Original: We have constructed a closed-form estimate for the KL term and show its gradients can be estimated by the reparameterization trick.
Revised: We have constructed a closed-form estimate for the KL term and show its gradients can be estimated by the reparameterization trick.

Original: Note that the reparameterization trick can also be used to compute the gradients of the reconstruction term in Eq.(1) and thus can be applied to the ELBO.
Revised: Note that the reparameterization trick can also be used to compute the gradients of the reconstruction term in Eq.(1) and thus can be applied to the ELBO.

Original: See Algo.for the complete algorithm.
Revised: See Algo.for the complete algorithm.

Original: Note that the number of samples ( M ) used in the reconstruction term is independent with those required for the KL estimation, which can be reduced when the model is expensive (e.g., we use only M = 1 in the experiments of VAEs).
Revised: Note that the number of samples ( M ) used in the reconstruction term is independent with those required for the KL estimation, which can be reduced when the model is expensive (e.g., we use only M = 1 in the experiments of VAEs).

Original: Thus the computational cost added compared to normal reparameterized VI is mainly the matrix inversion of size n p in Eq.(6).
Revised: Thus the computational cost added compared to normal reparameterized VI is mainly the matrix inversion of size n p in Eq.(6).

Original: In experiments we found tens or a hundred samples are enough to obtain an accurate KL estimate, so the added cost is not high.
Revised: In experiments we found tens or a hundred samples are enough to obtain an accurate KL estimate, so the added cost is not high.

Original: Require:
Revised: Require:

Original: Observed data x , model p θ ( x | z ) p ( z )
Revised: Observed data x , model p θ ( x | z ) p ( z )

Original: Require:
Revised: Require:

Original: Implicit variational posterior q φ ( z | x ) , n p , n q , M .
Revised: Implicit variational posterior q φ ( z | x ) , n p , n q , M .

Original: 1: repeat 2:
Revised: 1: repeat 2:

Original: Sample from prior: z pi ∼ p ( z ) , i = 1 , . .
Revised: Sample from prior: z pi ∼ p ( z ) , i = 1 , . .

Original: . , n p 3:
Revised: . , n p 3:

Original: Sample from variational: z qj ∼ q ( z | x ) , j = 1 , . .
Revised: Sample from variational: z qj ∼ q ( z | x ) , j = 1 , . .

Original: . , n q 4:
Revised: . , n q 4:

Original: Compute the density ratio ˆ r pq by Eq.
Revised: Compute the density ratio ˆ r pq by Eq.

Original: (5), (6) and clip ˆ r pq to be positive at z q s.
Revised: (5), (6) and clip ˆ r pq to be positive at z q s.

Original: Compute ˆKL = − 1 n q (cid:80) n q j =1 log ˆ r pq ( z qj ) 6:
Revised: Compute ˆKL = − 1 n q (cid:80) n q j =1 log ˆ r pq ( z qj ) 6:

Original: Compute L = 1 M (cid:80)
Revised: Compute L = 1 M (cid:80)

Original: Mm =1 p ( x | z qm ) − ˆKL and the gradient ∇ φ L with the reparameterization trick.
Revised: Mm =1 p ( x | z qm ) − ˆKL and the gradient ∇ φ L with the reparameterization trick.

Original: Do gradient descent with ∇ φ L 8: (Optional)
Revised: Do gradient descent with ∇ φ L 8: (Optional)

Original: For parameter learning, do gradient descent with ∇ θ L 9: until Convergence
Revised: For parameter learning, do gradient descent with ∇ θ L 9: until Convergence

Original: KIVI addresses the two challenges stated in Sec. 2.
Revised: KIVI addresses the two challenges stated in Sec. 2.

Original: First, the ratio estimates are given in closedforms, thus not having the problem of not catching up.
Revised: First, the ratio estimates are given in closedforms, thus not having the problem of not catching up.

Original: Second, the bias/variance trade-off of the estimation can be controlled by the regularization coefﬁcient λ .
Revised: Second, the bias/variance trade-off of the estimation can be controlled by the regularization coefﬁcient λ .

Original: When λ is set smaller, the estimation is more aggressive to match the samples.
Revised: When λ is set smaller, the estimation is more aggressive to match the samples.

Original: When λ is set larger, the estimated ratio function is smoother.
Revised: When λ is set larger, the estimated ratio function is smoother.

Original: Choosing the appropriate λ , the variance of estimation can be controlled while maintaining a reasonably good ﬁt, compared to the extreme ratio estimates given by classiﬁers when their probabilities are near 0 or 1.
Revised: Choosing the appropriate λ , the variance of estimation can be controlled while maintaining a reasonably good ﬁt, compared to the extreme ratio estimates given by classiﬁers when their probabilities are near 0 or 1.

Original: Moreover, KIVI is directly applicable to both global and local latent variable models (LVMs), which is an advantage over nonparametric VI methods like particle mirror descent (Dai et al., 2015) and stein variational gradient descent (Liu & Wang, 2016).
Revised: Moreover, KIVI is directly applicable to both global and local latent variable models (LVMs), which is an advantage over nonparametric VI methods like particle mirror descent (Dai et al., 2015) and stein variational gradient descent (Liu & Wang, 2016).

Original: For the task of training local
Revised: For the task of training local

Original: LVMs like VAEs, we we additionally use the adaptive contrast (AC) technique (Mescheder et al., 2017), whose details are summarized in Appendix D.
Revised: LVMs like VAEs, we we additionally use the adaptive contrast (AC) technique (Mescheder et al., 2017), whose details are summarized in Appendix D.

Original:  E XAMPLE : I MPLICIT V ARIATIONAL B AYESIAN N EURAL N ETWORKS
Revised:  E XAMPLE : I MPLICIT V ARIATIONAL B AYESIAN N EURAL N ETWORKS

Original: Now we present an example for using KIVI in Bayesian neural networks (BNNs), which has received increasing attention due to their ability to model uncertainty, an important factor in many tasks such as adversarial defense and reinforcement learning.
Revised: Now we present an example for using KIVI in Bayesian neural networks (BNNs), which has received increasing attention due to their ability to model uncertainty, an important factor in many tasks such as adversarial defense and reinforcement learning.

Original: However, it is nontrivial to apply implicit VI to BNNs because of the high-dimensional latent variables (weights).
Revised: However, it is nontrivial to apply implicit VI to BNNs because of the high-dimensional latent variables (weights).

Original: The recently developed implicit posteriors (Mescheder et al., 2017; Song et al., 2017) are not applicable.
Revised: The recently developed implicit posteriors (Mescheder et al., 2017; Song et al., 2017) are not applicable.

Original: We present Matrix Multiplication
Revised: We present Matrix Multiplication

Original: Neural Network (MMNN), an efﬁcient framework for sampling large matrices.
Revised: Neural Network (MMNN), an efﬁcient framework for sampling large matrices.

Original: Deploying MMNN,
Revised: Deploying MMNN,

Original: KIVI can easily scale up to large BNNs.
Revised: KIVI can easily scale up to large BNNs.

Original: In BNNs, a prior is speciﬁed over the neural network parameters W = { W l } Ll =1 , where W l indicates weights in the l -th layer.
Revised: In BNNs, a prior is speciﬁed over the neural network parameters W = { W l } Ll =1 , where W l indicates weights in the l -th layer.

Original: Given input x , the output y is modeled with
Revised: Given input x , the output y is modeled with

Original: [Equation]
Revised: [Equation]

Original: where ˆ y is the output of the feed-forward network f NN .
Revised: where ˆ y is the output of the feed-forward network f NN .

Original: And y is of a distribution P parameterized by ˆ y and θ .
Revised: And y is of a distribution P parameterized by ˆ y and θ .

Original: For regression, P is usually a Gaussian with ˆ y as the mean.
Revised: For regression, P is usually a Gaussian with ˆ y as the mean.

Original: For classiﬁcation, P is usually a discrete distribution with ˆ y as the unnormalized log probabilities.
Revised: For classiﬁcation, P is usually a discrete distribution with ˆ y as the unnormalized log probabilities.

Original: The true posterior of W in BNNs is intractable.
Revised: The true posterior of W in BNNs is intractable.

Original: Thus we turn to VI and use a variational posterior q to approximate it.
Revised: Thus we turn to VI and use a variational posterior q to approximate it.

Original: Denoting the dataset with X = { x i } Ni =1 , Y = { y i } Ni =1 , we have the ELBO:
Revised: Denoting the dataset with X = { x i } Ni =1 , Y = { y i } Ni =1 , we have the ELBO:

Original: [Equation]
Revised: [Equation]

Original: The variational posterior is usually set to be factorized by layer: q φ ( W ) = (cid:81)
Revised: The variational posterior is usually set to be factorized by layer: q φ ( W ) = (cid:81)

Original: Ll =
Revised: Ll =

Original: 1 q φ l ( W l ) .
Revised: 1 q φ l ( W l ) .

Original: However, previous methods used variational posteriors with a limited capacity for each q φ l ( W l ) , including factorized Gaussian (Hernandez-Lobato & Adams, 2015), matrix variate Gaussian (Louizos &
Revised: However, previous methods used variational posteriors with a limited capacity for each q φ l ( W l ) , including factorized Gaussian (Hernandez-Lobato & Adams, 2015), matrix variate Gaussian (Louizos &

Original: Welling, 2016; Sun et al., 2017) and normalizing ﬂows (Louizos & Welling, 2017).
Revised: Welling, 2016; Sun et al., 2017) and normalizing ﬂows (Louizos & Welling, 2017).

Original: Enabled to learn implicit variational posterior, we propose to adopt a general distribution without an explicit density function, which has a form of
Revised: Enabled to learn implicit variational posterior, we propose to adopt a general distribution without an explicit density function, which has a form of

Original: [Equation]
Revised: [Equation]

Original: Require:
Revised: Require:

Original: Input matrix X
Revised: Input matrix X

Original: Require: network parameters { W li , B li , W ri , B ri } Li = 1: for i = 1 : L do 2:
Revised: Require: network parameters { W li , B li , W ri , B ri } Li = 1: for i = 1 : L do 2:

Original: Here g is a transform parameterized by φ l .
Revised: Here g is a transform parameterized by φ l .

Original: The key challenge here is that W l are very high dimensional for moderate size neural networks.
Revised: The key challenge here is that W l are very high dimensional for moderate size neural networks.

Original: Thus, we often cannot use a fully connected neural network (MLP) as g here.
Revised: Thus, we often cannot use a fully connected neural network (MLP) as g here.

Original: Inspired by low-rank matrix factorization (Koren et al., 2009), we propose a new kind of network called Matrix Multiplication Neural Network (MMNN) to serve as the g function here, as shown in Alg. 2.
Revised: Inspired by low-rank matrix factorization (Koren et al., 2009), we propose a new kind of network called Matrix Multiplication Neural Network (MMNN) to serve as the g function here, as shown in Alg. 2.

Original: In each layer of a MMNN, for input matrix
Revised: In each layer of a MMNN, for input matrix

Original: X 

Revised: X

Original: ( M in × N in ), we left multiply a parameter matrix W li ( M out × M in ) and add a bias matrix B li ( M out × N in ), then we right multiply a parameter matrix ( N in × N out ) and add a bias matrix ( M out × N out ).
Revised: ( M in × N in ), we left multiply a parameter matrix W li ( M out × M in ) and add a bias matrix B li ( M out × N in ), then we right multiply a parameter matrix ( N in × N out ) and add a bias matrix ( M out × N out ).

Original: Finally it is passed through a nonlinear activation function such as ReLU.
Revised: Finally it is passed through a nonlinear activation function such as ReLU.

Original: We call such a layer as Matrix Multiplication Layer of size M out × N out .
Revised: We call such a layer as Matrix Multiplication Layer of size M out × N out .

Original: When modeling a matrix, MMNN has signiﬁcant computational advantages over MLPs.
Revised: When modeling a matrix, MMNN has signiﬁcant computational advantages over MLPs.

Original: Due to its low-rank property, MMNN easily scales with the matrix size.
Revised: Due to its low-rank property, MMNN easily scales with the matrix size.

Original: To model a M × N matrix, we consider a one-layer network whose input shape is M 0 × N 0 .
Revised: To model a M × N matrix, we consider a one-layer network whose input shape is M 0 × N 0 .

Original: For the fully connected structure, the parameter matrix’s shape is M 0 N 0 × MN .
Revised: For the fully connected structure, the parameter matrix’s shape is M 0 N 0 × MN .

Original: While for MMNN structure, we only need two matrices of shape
Revised: While for MMNN structure, we only need two matrices of shape

Original: M × M 0 and two matrices of shape N 0 × N .
Revised: M × M 0 and two matrices of shape N 0 × N .

Original: To model the implicit posterior of W l , we only need to randomly sample a matrix W 0 l of smaller size M 0 × N 0 , and feed it forward through the MMNN to get the output variational samples ( W ql ):
Revised: To model the implicit posterior of W l , we only need to randomly sample a matrix W 0 l of smaller size M 0 × N 0 , and feed it forward through the MMNN to get the output variational samples ( W ql ):

Original: [Equation]
Revised: [Equation]

Original: Thus, we can use a MMNN to model the variational posterior of each layer’s parameter W l for large scale networks.
Revised: Thus, we can use a MMNN to model the variational posterior of each layer’s parameter W l for large scale networks.

Original: However, in tasks with small networks, we still use MLPs to model q .
Revised: However, in tasks with small networks, we still use MLPs to model q .

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Our work closely relates to implicit generative models (IGMs) and variational inference (VI).
Revised: Our work closely relates to implicit generative models (IGMs) and variational inference (VI).

Original: IGMs (generative models that deﬁne implicit distributions) have drawn much attention due to the popularity of GAN.
Revised: IGMs (generative models that deﬁne implicit distributions) have drawn much attention due to the popularity of GAN.

Original: General learning algorithms of implicit models have been surveyed in Mohamed &
Revised: General learning algorithms of implicit models have been surveyed in Mohamed &

Original: Lakshminarayanan (2016), of which density ratio estimation plays the central role.
Revised: Lakshminarayanan (2016), of which density ratio estimation plays the central role.

Original: The connection between density ratio estimation and GANs is also discussed in (Uehara et al., 2016).
Revised: The connection between density ratio estimation and GANs is also discussed in (Uehara et al., 2016).

Original: Our work also builds upon the recent VI methods, including stochastic optimization by mini-batches (Hoffman et al., 2013), direct gradient optimization of variational lower bounds (Paisley et al.,
Revised: Our work also builds upon the recent VI methods, including stochastic optimization by mini-batches (Hoffman et al., 2013), direct gradient optimization of variational lower bounds (Paisley et al.,

Original: [Table]
Revised: [Table]

Original: Year
Revised: Year

Original: [Table]
Revised: [Table]

Original: 2012; Mnih & Gregor, 2014), and the reparametrization trick for continuous LVMs (Kingma &
Revised: 2012; Mnih & Gregor, 2014), and the reparametrization trick for continuous LVMs (Kingma &

Original: Welling, 2013).
Revised: Welling, 2013).

Original: Following the success of learning with IGMs, implicit distributions are applied to
Revised: Following the success of learning with IGMs, implicit distributions are applied to

Original: VI.
Revised: VI.

Original: Husz´ar (2017) divide implicit VI using a discriminator into two categories: prior-contrastive and joint-contrastive.
Revised: Husz´ar (2017) divide implicit VI using a discriminator into two categories: prior-contrastive and joint-contrastive.

Original: Classiﬁers in prior-contrastive methods distinguish between samples from the prior and the variational posterior, while in joint-contrastive methods they distinguish between the model joint distribution and the joint distribution composed of data distribution and variational posteriors.
Revised: Classiﬁers in prior-contrastive methods distinguish between samples from the prior and the variational posterior, while in joint-contrastive methods they distinguish between the model joint distribution and the joint distribution composed of data distribution and variational posteriors.

Original: Concurrent with Husz´ar (2017), Mescheder et al.
Revised: Concurrent with Husz´ar (2017), Mescheder et al.

Original: (2017) proposed Adversarial Variational
Revised: (2017) proposed Adversarial Variational

Original: Bayes (AVB), which is an amortized version of prior-contrastive methods for training local LVMs like VAEs (Kingma & Welling, 2013).
Revised: Bayes (AVB), which is an amortized version of prior-contrastive methods for training local LVMs like VAEs (Kingma & Welling, 2013).

Original: Prior to Husz ´ ar (2017), similar ideas with joint-contrastive methods have been proposed in ALI and Bi-GAN (Dumoulin et al., 2016; Donahue et al., 2016).
Revised: Prior to Husz ´ ar (2017), similar ideas with joint-contrastive methods have been proposed in ALI and Bi-GAN (Dumoulin et al., 2016; Donahue et al., 2016).

Original: Nonparametric methods for VI (Liu & Wang, 2016; Dai et al., 2015) that adapt a set of particles towards the true posterior are also closely related to implicit VI.
Revised: Nonparametric methods for VI (Liu & Wang, 2016; Dai et al., 2015) that adapt a set of particles towards the true posterior are also closely related to implicit VI.

Original: They share the similar advantage of ﬂexible approximations.
Revised: They share the similar advantage of ﬂexible approximations.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We present empirical results on both synthetic and real datasets to demonstrate the beneﬁts of KIVI.
Revised: We present empirical results on both synthetic and real datasets to demonstrate the beneﬁts of KIVI.

Original: 6. T OY 1-D G AUSSIAN M IXTURES
Revised: 6. T OY 1-D G AUSSIAN M IXTURES

Original: [Figure]
Revised: [Figure]

Original: We ﬁrstly conduct a toy experiment to approximate a 1-D Gaussian mixture distribution with VI.
Revised: We ﬁrstly conduct a toy experiment to approximate a 1-D Gaussian mixture distribution with VI.

Original: The Gaussian mixture distribution has two equally distributed unit-variance components whose means are -3 and 3.
Revised: The Gaussian mixture distribution has two equally distributed unit-variance components whose means are -3 and 3.

Original: We compare
Revised: We compare

Original: KIVI with VI using a single Gaussian posterior (Fig. 1).
Revised: KIVI with VI using a single Gaussian posterior (Fig. 1).

Original: The variational distribution used by
Revised: The variational distribution used by

Original: KIVI generates samples by forwarding a standard normal distribution through a two-layer
Revised: KIVI generates samples by forwarding a standard normal distribution through a two-layer

Original: MLP with 10 hidden units each and one output unit.
Revised: MLP with 10 hidden units each and one output unit.

Original: As shown, the Gaussian posterior converges to the middle of the two modes, where the probability mass is small.
Revised: As shown, the Gaussian posterior converges to the middle of the two modes, where the probability mass is small.

Original: In contrast, KIVI can accurately approximate the Gaussian mixture distribution with an expressive variational posterior.
Revised: In contrast, KIVI can accurately approximate the Gaussian mixture distribution with an expressive variational posterior.

Original: 6. B AYESIAN N EURAL N ETWORKS (BNN S )
Revised: 6. B AYESIAN N EURAL N ETWORKS (BNN S )

Original: As stated in Sec. 4, the latent variables in BNNs are global to all data points and are usually very high-dimensional, for which a ﬂexible variational family is essential.
Revised: As stated in Sec. 4, the latent variables in BNNs are global to all data points and are usually very high-dimensional, for which a ﬂexible variational family is essential.

Original: We compare KIVI with state-of-the-art VI methods by doing regression and classiﬁcation on standard benchmarks.
Revised: We compare KIVI with state-of-the-art VI methods by doing regression and classiﬁcation on standard benchmarks.

Original: 6.2. R EGRESSION
Revised: 6.2. R EGRESSION

Original: To quantitatively measure the predictive ability of BNNs with KIVI as the inference method, we use standard multivariate regression benchmarks from recent works (Table 1), such as probabilistic backprogation (PBP) (Hernandez-Lobato & Adams, 2015).
Revised: To quantitatively measure the predictive ability of BNNs with KIVI as the inference method, we use standard multivariate regression benchmarks from recent works (Table 1), such as probabilistic backprogation (PBP) (Hernandez-Lobato & Adams, 2015).

Original: We compare with state-of-the-art methods: the Bayesian interpretation of dropout (Gal & Ghahramani, 2016) and stein variational gradient
Revised: We compare with state-of-the-art methods: the Bayesian interpretation of dropout (Gal & Ghahramani, 2016) and stein variational gradient

Original: [Figure]
Revised: [Figure]

Original: descent (SVGD) (Liu & Wang, 2016) 1 .
Revised: descent (SVGD) (Liu & Wang, 2016) 1 .

Original: Following the setup in PBP
Revised: Following the setup in PBP

Original: [Figure]
Revised: [Figure]

Original: we use BNNs with one 50-unit hidden layer except in the two large datasets, i
Revised: we use BNNs with one 50-unit hidden layer except in the two large datasets, i

Original: [Figure]
Revised: [Figure]

Original: e., Protein Structure and Year Predication MSD , where 100 units are used.
Revised: e., Protein Structure and Year Predication MSD , where 100 units are used.

Original: We randomly select 90% of the whole dataset for training and the rest for testing.
Revised: We randomly select 90% of the whole dataset for training and the rest for testing.

Original: We also put a Gamma prior on the observation noise precision to adaptively learn it (see Appendix G).
Revised: We also put a Gamma prior on the observation noise precision to adaptively learn it (see Appendix E).

Original: For all datasets, we use batch size 100, n p = n q = M = 100 , λ = 0 .
Revised: For all datasets, we use batch size 100, n p = n q = M = 100 , λ = 0 .

Original: 001 , learning rate 1e-3.
Revised: 001 , learning rate 1e-3.

Original: The model is trained for 3000 epochs for small datasets with less than 1000 data points, and 500 epochs for others.
Revised: The model is trained for 3000 epochs for small datasets with less than 1000 data points, and 500 epochs for others.

Original: We report the mean errors and standard deviations averaged over 20 runs, except 5 runs for
Revised: We report the mean errors and standard deviations averaged over 20 runs, except 5 runs for

Original: Protein Structure and 1 run for Year Predication MSD .
Revised: Protein Structure and 1 run for Year Predication MSD .

Original: As networks used in these tasks are of small scale, we use MLPs with one hidden layer in the implicit variational posterior (see Appendix F.3.
Revised: As networks used in these tasks are of small scale, we use MLPs with one hidden layer in the implicit variational posterior (see Appendix G.3.

Original: for details).
Revised: for details).

Original: Table 1 shows the results with the best ones marked in bold.
Revised: Table 1 shows the results with the best ones marked in bold.

Original: Results of SVGD and dropout are copied from their papers, which have the same setting with us.
Revised: Results of SVGD and dropout are copied from their papers, which have the same setting with us.

Original: We can see that KIVI consistently outperforms SVGD and dropout on both RMSE and test-LL for most datasets.
Revised: We can see that KIVI consistently outperforms SVGD and dropout on both RMSE and test-LL for most datasets.

Original: Especially on RMSE,
Revised: Especially on RMSE,

Original: KIVI has signiﬁcant improvements over them except on Wine and Year Predication MSD .
Revised: KIVI has signiﬁcant improvements over them except on Wine and Year Predication MSD .

Original: It suggests that KIVI enables the implicit variational posterior to capture the structured uncertainty in network parameters, which is hard to be fully described by a mixture of two delta distributions (dropout) and a ﬁxed set of particles (SVGD).
Revised: It suggests that KIVI enables the implicit variational posterior to capture the structured uncertainty in network parameters, which is hard to be fully described by a mixture of two delta distributions (dropout) and a ﬁxed set of particles (SVGD).

Original: We emphasize that although the nonparametric nature of SVGD has also made the approximation more ﬂexible, it uses the same set of particles throughout the inference produre, while each iteration of KIVI generates a new set of particles.
Revised: We emphasize that although the nonparametric nature of SVGD has also made the approximation more ﬂexible, it uses the same set of particles throughout the inference produre, while each iteration of KIVI generates a new set of particles.

Original: Thus the implicit posterior learned by KIVI is smoothed by the parametric model.
Revised: Thus the implicit posterior learned by KIVI is smoothed by the parametric model.

Original: Recently, normalizing ﬂows have shown good performance on BNNs (Louizos & Welling, 2017).
Revised: Recently, normalizing ﬂows have shown good performance on BNNs (Louizos & Welling, 2017).

Original: So we also experiment with directly applying normalizing ﬂows on this task.
Revised: So we also experiment with directly applying normalizing ﬂows on this task.

Original: The results are reported in Appendix E.2.
Revised: The results are reported in Appendix F.2.

Original: 6.2. C LASSIFICATION
Revised: 6.2. C LASSIFICATION

Original: We ﬁnally present the classiﬁcation results on MNIST, which consists of 60k training images and 10k test images of handwriting digits.
Revised: We ﬁnally present the classiﬁcation results on MNIST, which consists of 60k training images and 10k test images of handwriting digits.

Original: Compared to the regression problems, MNIST has a much higher feature dimension, introducing millions of parameters even in moderate-size networks.
Revised: Compared to the regression problems, MNIST has a much higher feature dimension, introducing millions of parameters even in moderate-size networks.

Original: Thus it brings big challenges for BNNs.
Revised: Thus it brings big challenges for BNNs.

Original: As a standard benchmark, the performance on MNIST can be improved by many other techniques: convolution, generative pretraining, data augmentation, etc.
Revised: As a standard benchmark, the performance on MNIST can be improved by many other techniques: convolution, generative pretraining, data augmentation, etc.

Original: To ensure fair comparison, we follow the settings from Bayes-By-Backprop (Blundell et al., 2015) and focus on improving the performance of ordinary MLPs without using any of these techniques.
Revised: To ensure fair comparison, we follow the settings from Bayes-By-Backprop (Blundell et al., 2015) and focus on improving the performance of ordinary MLPs without using any of these techniques.

Original: The network structures used are three MLPs, all with two ReLU hidden layers, and the layer sizes are 400,and 1200, respectively.
Revised: The network structures used are three MLPs, all with two ReLU hidden layers, and the layer sizes are 400,and 1200, respectively.

Original: For KIVI, we used MMNNs with two hidden matrix multiplication layers in the implicit posterior (see Appendix F.3.2 for details).
Revised: For KIVI, we used MMNNs with two hidden matrix multiplication layers in the implicit posterior (see Appendix G.3.2 for details).

Original: We set n p = n q = M = 10 , λ = 0 .
Revised: We set n p = n q = M = 10 , λ = 0 .

Original: 001 , batch size 100 and trained for 300 epochs.
Revised: 001 , batch size 100 and trained for 300 epochs.

Original: The initial learning rate is 0.001 and is annealed every
Revised: The initial learning rate is 0.001 and is annealed every

Original: [Figure]
Revised: [Figure]

Original: epochs by ratio 0.5.
Revised: epochs by ratio 0.5.

Original: We used the last 10k samples of the training set as the validation set for model selection.
Revised: We used the last 10k samples of the training set as the validation set for model selection.

Original: Fig.
Revised: Fig.

Original: 2 summarizes the results.
Revised: 2 summarizes the results.

Original: We can see that KIVI achieves better accuracy compared to plain SGD (Simard et al., 2003), dropout (Srivastava et al., 2014) and Bayes-By-Backprop (Blundell et al., 2015) on all three types of MLPs.
Revised: We can see that KIVI achieves better accuracy compared to plain SGD (Simard et al., 2003), dropout (Srivastava et al., 2014) and Bayes-By-Backprop (Blundell et al., 2015) on all three types of MLPs.

Original: KIVI even performs better than Bayes-By-Backprop with a changed prior (scale mixture), which makes the model iterself more ﬂexible than ours.
Revised: KIVI even performs better than Bayes-By-Backprop with a changed prior (scale mixture), which makes the model iterself more ﬂexible than ours.

Original: When the layer size is 800, our result is comparable to an ensemble of 5 networks with dropconnect (Wan et al., 2013), which demonstrates that the implicit posterior has been learned to account for most model uncertainty.
Revised: When the layer size is 800, our result is comparable to an ensemble of 5 networks with dropconnect (Wan et al., 2013), which demonstrates that the implicit posterior has been learned to account for most model uncertainty.

Original: We also conduct experiments with the prior-contrastive (PC) method (the corresponding counterpart of AVB for global latent variables, see Sec. 2).
Revised: We also conduct experiments with the prior-contrastive (PC) method (the corresponding counterpart of AVB for global latent variables, see Sec. 2).

Original: The key challenge for applying PC here is that posterior samples are extremely high-dimensional, and if fed into discriminators like neural networks, they will cause unaffordable computation cost.
Revised: The key challenge for applying PC here is that posterior samples are extremely high-dimensional, and if fed into discriminators like neural networks, they will cause unaffordable computation cost.

Original: To get around this, we use logistic regression as the discriminator in PC.
Revised: To get around this, we use logistic regression as the discriminator in PC.

Original: The experiment settings of PC are reported in Appendix F.3.2.
Revised: The experiment settings of PC are reported in Appendix G.3.2.

Original: The training lower bounds of the two methods are plotted in Fig.
Revised: The training lower bounds of the two methods are plotted in Fig.

Original: We can see that at ﬁrst they increase at the same pace, then PC fails to converge with lower bound explosion while KIVI improves consistently.
Revised: We can see that at ﬁrst they increase at the same pace, then PC fails to converge with lower bound explosion while KIVI improves consistently.

Original: The explosion is mainly because the input to the discriminator is of hundreds of thousands of dimensions, and plain logistic regression cannot produce reliable density ratio estimates.
Revised: The explosion is mainly because the input to the discriminator is of hundreds of thousands of dimensions, and plain logistic regression cannot produce reliable density ratio estimates.

Original: We also experiment with
Revised: We also experiment with

Original: PC for layer size 800 and 1200.
Revised: PC for layer size 800 and 1200.

Original: They both fail to converge in the end.
Revised: They both fail to converge in the end.

Original: 6. V ARIATIONAL A UTOENCODERS
Revised: 6. V ARIATIONAL A UTOENCODERS

Original: As stated in Sec. 3.2, KIVI is applicable to both global and local LVMs, and the latter can be learned using an amortized scheme (i.e., use q φ ( z | x ) instead of q φ ( z ) ).
Revised: As stated in Sec. 3.2, KIVI is applicable to both global and local LVMs, and the latter can be learned using an amortized scheme (i.e., use q φ ( z | x ) instead of q φ ( z ) ).

Original: Here we present an application on training variational autoencoders (VAE) with implicit posteriors to demonstrate this.
Revised: Here we present an application on training variational autoencoders (VAE) with implicit posteriors to demonstrate this.

Original: The generative process of VAEs proceeds as
Revised: The generative process of VAEs proceeds as

Original: [Equation]
Revised: [Equation]

Original: where z are latent features, x are observations, P is the output distribution for modeling the observations, which takes the outputs of the neural network ( f NN ( z ) ) as parameters.
Revised: where z are latent features, x are observations, P is the output distribution for modeling the observations, which takes the outputs of the neural network ( f NN ( z ) ) as parameters.

Original: We conduct experiments on two widely used datasets for generative modeling: binarized MNIST and CelebA (Liu et al., 2015).
Revised: We conduct experiments on two widely used datasets for generative modeling: binarized MNIST and CelebA (Liu et al., 2015).

Original: P is Bernoulli for binarized MNIST, while a normal distribution for CelebA. For VAE the variational lower bound is the same form as Eq.
Revised: P is Bernoulli for binarized MNIST, while a normal distribution for CelebA. For VAE the variational lower bound is the same form as Eq.

Original: (1), except q φ ( z ) is replaced by q φ ( z | x ) .
Revised: (1), except q φ ( z ) is replaced by q φ ( z | x ) .

Original: The original VAE parameterizes q φ ( z | x ) as illustrated in Fig.
Revised: The original VAE parameterizes q φ ( z | x ) as illustrated in Fig.

Original: 3a (left).
Revised: 3a (left).

Original: To form an implicit posterior, a direct choice is to move the stochastic noise (denoted as z i in the ﬁgure) one layer down the network and add a fully-connected layer on top of it to produce samples, as illustrated by Fig.
Revised: To form an implicit posterior, a direct choice is to move the stochastic noise (denoted as z i in the ﬁgure) one layer down the network and add a fully-connected layer on top of it to produce samples, as illustrated by Fig.

Original: 3a (right).
Revised: 3a (right).

Original: Before applying KIVI, a crucial question is what we expect to get by using implicit posteriors for training VAEs.
Revised: Before applying KIVI, a crucial question is what we expect to get by using implicit posteriors for training VAEs.

Original: One target could be that we may get tighter lower bounds of the data log-likelihood (LL) because the algorithm searches in a larger variational family for the optimal lower bound.
Revised: One target could be that we may get tighter lower bounds of the data log-likelihood (LL) because the algorithm searches in a larger variational family for the optimal lower bound.

Original: This suggests, however in a very weak way, that optimizing in the larger space will lead to better test
Revised: This suggests, however in a very weak way, that optimizing in the larger space will lead to better test

Original: LL, given the optimization always arrives at local optima.
Revised: LL, given the optimization always arrives at local optima.

Original: Previously Adversarial Variational Bayes (AVB) has shown some results on MNIST, by comparing the test LL of models trained by plain VAE and AVB, using golden truths estimated by annealed importance sampling (AIS)
Revised: Previously Adversarial Variational Bayes (AVB) has shown some results on MNIST, by comparing the test LL of models trained by plain VAE and AVB, using golden truths estimated by annealed importance sampling (AIS)

Original: (Wu et al., 2016).
Revised: (Wu et al., 2016).

Original: However, for the results reported in AVB, the model architectures used by plain VAE and AVB are very different, which leads to concerns about which part of the change contributes to the improved likelihoods.
Revised: However, for the results reported in AVB, the model architectures used by plain VAE and AVB are very different, which leads to concerns about which part of the change contributes to the improved likelihoods.

Original: Here we adopt another setting to better demonstrate the gain from implicit posteriors.
Revised: Here we adopt another setting to better demonstrate the gain from implicit posteriors.

Original: We observe the key improvements of implicit posteriors is that objectives with them average over a much broader range of posterior conﬁgurations.
Revised: We observe the key improvements of implicit posteriors is that objectives with them average over a much broader range of posterior conﬁgurations.

Original: This effect not only contributes to a larger search space that contains tighter lower bound values, but also keeps the VAE model better prevent overﬁtting.
Revised: This effect not only contributes to a larger search space that contains tighter lower bound values, but also keeps the VAE model better prevent overﬁtting.

Original: To verify that
Revised: To verify that

Original: KIVI keeps this property, we conduct experiments on a statically binarized MNIST dataset and use models with no prior knowledge of the problem (MLP instead of convnets), which is a typical setting that leads to overﬁtting.
Revised: KIVI keeps this property, we conduct experiments on a statically binarized MNIST dataset and use models with no prior knowledge of the problem (MLP instead of convnets), which is a typical setting that leads to overﬁtting.

Original: The  latent dimension  of  the used VAE model is 8, more details can be found in Appendix F.2.
Revised: The VAE model has latent dimension 8, f NN ( z ) is a MLP with two hidden layers of size 500, and the parameters for KIVI are n p = n q = 100 , M = 1 , more details can be found in Appendix G.2.

Original: Fig 3b shows the training curves of VAEs with or without KIVI.
Revised: Fig 3b shows the training curves of VAEs with or without KIVI.

Original: We can see that the plain VAEs quickly overﬁt the training set while the KIVI trained VAEs are more robust against overﬁtting.
Revised: We can see that the plain VAEs quickly overﬁt the training set while the KIVI trained VAEs are more robust against overﬁtting.

Original: After 3k epochs we evaluate the test LL on 2048 test images using AIS and get -97.3 for plain VAEs, while -94.8 for KIVI trained VAEs.
Revised: After 3k epochs we evaluate the test LL on 2048 test images using AIS and get -97.3 for plain VAEs, while -94.8 for KIVI trained VAEs.

Original: To demonstrate that KIVI scales to larger models, we trained a VAE with the same network structure used by DCGAN (Radford et al., 2015) on CelebA using KIVI, the latent dimimension in this case are 32.
Revised: To demonstrate that KIVI scales to larger models, we trained a VAE with the same network structure used by DCGAN (Radford et al., 2015) on CelebA using KIVI, the latent dimimension in this case are 32.

Original: The implicit posterior is constructed in the way shown in Fig.
Revised: The implicit posterior is constructed in the way shown in Fig.

Original: 3a, with the bottom hidden layers symmetric to the decoder network (See Appendix F.2 for details).
Revised: 3a, with the bottom hidden layers symmetric to the decoder network (See Appendix G.2 for details).

Original: To visually check the latent space learned by KIVI, we show the reconstruction results of linearly interpolating between the latent z vector of two real images in Fig.
Revised: To visually check the latent space learned by KIVI, we show the reconstruction results of linearly interpolating between the latent z vector of two real images in Fig.

Original: 4 after 25 epochs, when the model converges.
Revised: 4 after 25 epochs, when the model converges.

Original: Compared to the latent space learned by AVB after the same epochs (we use the public code from AVB and set the same decoder structure), we ﬁnd ours are smoother, and the interpolated images are much sharper.
Revised: Compared to the latent space learned by AVB after the same epochs (we use the public code from AVB and set the same decoder structure), we ﬁnd ours are smoother, and the interpolated images are much sharper.

Revised: More interpolation results through the training process are presented and compared in Appendix F.

Original: In fact many efforts are required to make AVB successfully train a model for CelebA to produce results shown in the ﬁgure.
Revised: In fact many efforts are required to make AVB successfully train a model for CelebA to produce results shown in the ﬁgure.

Original: As reported in Mescheder et al.
Revised: As reported in Mescheder et al.

Original: (2017), the log prior log p ( z ) is explicitly added to the discriminator ( T ( x , z ) ), while KIVI doesn’t need much tuning: there is no need to carefully design the discriminator, and the only two hyper-parameters (i.e, λ and the clipping value) have clear meanings.
Revised: (2017), the log prior log p ( z ) is explicitly added to the discriminator ( T ( x , z ) ), while KIVI doesn’t need much tuning: there is no need to carefully design the discriminator, and the only two hyper-parameters (i.e, λ and the clipping value) have clear meanings.

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: We present an implicit variational inference method named Kernel Implicit Variational Inference (KIVI), which addresses the existing challenges of implicit variational inference, including noisy estimation and scalability with high-dimensional latent variable models.
Revised: We present an implicit variational inference method named Kernel Implicit Variational Inference (KIVI), which addresses the existing challenges of implicit variational inference, including noisy estimation and scalability with high-dimensional latent variable models.

Original: We successfully apply this approach to Bayesian neural networks and achieve superior performance on both regression and classiﬁcation tasks.
Revised: We successfully apply this approach to Bayesian neural networks and achieve superior performance on both regression and classiﬁcation tasks.

Original: We also demonstrate that KIVI can be applied to learn local latent variable models like VAEs with implicit posteriors successfully.
Revised: We also demonstrate that KIVI can be applied to learn local latent variable models like VAEs with implicit posteriors successfully.

Original: Future work may include applying this method on neural networks with larger scale and improving the kernel estimator further.
Revised: Future work may include applying this method on neural networks with larger scale and improving the kernel estimator further.

Original: A P ROOF OF P ROPOSITION
Revised: A P ROOF OF P ROPOSITION

Original: Proof.
Revised: Proof.

Original: Let V denote the span of the representers of the sample points:
Revised: Let V denote the span of the representers of the sample points:

Original: [Equation]
Revised: [Equation]

Original: where we label z p 1: n p as z 1: n p and z q 1: n q as z n p : n p + n q .
Revised: where we label z p 1: n p as z 1: n p and z q 1: n q as z n p : n p + n q .

Original: Deﬁne the orthogonal complement V to be
Revised: Deﬁne the orthogonal complement V to be

Original: [Equation]
Revised: [Equation]

Original: Because ˆ r ∈ H , it can be decomposed into two parts:
Revised: Because ˆ r ∈ H , it can be decomposed into two parts:

Original: [Equation]
Revised: [Equation]

Original: where ˆ r (cid:107) ∈ V , and ˆ r ⊥ ∈ V ⊥ .
Revised: where ˆ r (cid:107) ∈ V , and ˆ r ⊥ ∈ V ⊥ .

Original: Note that the squared loss is not changed by the orthogonal component ˆ r ⊥ :
Revised: Note that the squared loss is not changed by the orthogonal component ˆ r ⊥ :

Original: [Equation]
Revised: [Equation]

Original: Meanwhile the regularization term can be decomposed into (cid:107) ˆ r (cid:107) (cid:107) 2 H +
Revised: Meanwhile the regularization term can be decomposed into (cid:107) ˆ r (cid:107) (cid:107) 2 H +

Original: [Equation]
Revised: [Equation]

Original: ˆ r ⊥ (cid:107) 2 H .
Revised: ˆ r ⊥ (cid:107) 2 H .

Original: So the optimal solution will have ˆ r ∗⊥ = 0 , which indicates that ˆ r ∗ ∈ V .
Revised: So the optimal solution will have ˆ r ∗⊥ = 0 , which indicates that ˆ r ∗ ∈ V .

Original: B D ERIVATION OF THE O PTIMAL D ENSITY R ATIO F UNCTION
Revised: B D ERIVATION OF THE O PTIMAL D ENSITY R ATIO F UNCTION

Original: Substituting Eq.
Revised: Substituting Eq.

Original: (5) into Eq.
Revised: (5) into Eq.

Original: (3) and removing the constant, we get
Revised: (3) and removing the constant, we get

Original: L [Equation]
Revised: L ( α , β ) = 1 2 n p (cid:0) (cid:107) K

Original:  α, β
Revised: p α

Original: [Equation]{+(cid:107) 
+}
Revised: {+(cid:107) 
+}

Original: + 
Revised: + (cid:107) K pq β (cid:107)

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Taking derivatives w
Revised: Taking derivatives w

Original: [Equation]
Revised: [Equation]

Original: r
Revised: r

Original: [Equation]
Revised: [Equation]

Original: t. α and β and setting them to be zeros:
Revised: t. α and β and setting them to be zeros:

Original: [Equation]
Revised: [Equation]

Original: Rearranging the terms, we have
Revised: Rearranging the terms, we have

Original: [Equation]
Revised: [Equation]

Original: Left multiplying Eq.
Revised: Left multiplying Eq.

Original: (20) with K (cid:62) pq K − 1 p and then subtracting Eq.
Revised: (20) with K (cid:62) pq K − 1 p and then subtracting Eq.

Original: (21) yields
Revised: (21) yields

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Substituting the optimal β into Eq.
Revised: Substituting the optimal β into Eq.

Original: ( 20), we get
Revised: ( 20), we get

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Note that although we use K − 1 p in the derivation, the forms of the solution do not require K p to be invertible.
Revised: Note that although we use K − 1 p in the derivation, the forms of the solution do not require K p to be invertible.

Original: In fact, if K p has zero eigenvalues (not invertible), the objective L ( α , β ) is not bounded below since one can always choose a α in the null space of K p to decrease it.
Revised: In fact, if K p has zero eigenvalues (not invertible), the objective L ( α , β ) is not bounded below since one can always choose a α in the null space of K p to decrease it.

Original: In other words, the form of the solution is well deﬁned in any condition and is optimal when the objective is well deﬁned.
Revised: In other words, the form of the solution is well deﬁned in any condition and is optimal when the objective is well deﬁned.

Original: C G RADIENTS OF THE KL T ERM
Revised: C G RADIENTS OF THE KL T ERM

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: The formula above shows that a good density ratio estimator gives accurate gradient estimation of φ .
Revised: The formula above shows that a good density ratio estimator gives accurate gradient estimation of φ .

Original: D KIVI WITH A DAPTIVE C ONTRAST
Revised: D KIVI WITH A DAPTIVE C ONTRAST

Original: Although KIVI gives rather good estimation of the density ratio, the estimation accuracy still degrades with larger discrepancy between p and q .
Revised: Although KIVI gives rather good estimation of the density ratio, the estimation accuracy still degrades with larger discrepancy between p and q .

Original: The problem is very critical in training local latent variable models like VAEs because the same variational model is required to infer posteriors of all local latent variables.
Revised: The problem is very critical in training local latent variable models like VAEs because the same variational model is required to infer posteriors of all local latent variables.

Original: In order to mitigate that, AVB (Mescheder et al., 2017) adopted a technique called
Revised: In order to mitigate that, AVB (Mescheder et al., 2017) adopted a technique called

Original: Adaptive Contrast (AC), which can easily be integrated with KIVI.
Revised: Adaptive Contrast (AC), which can easily be integrated with KIVI.

Original: AC introduces an auxiliary explicit distribution r α ( z | x ) that resembles q .
Revised: AC introduces an auxiliary explicit distribution r α ( z | x ) that resembles q .

Original: With r α
Revised: With r α

Original: ( z | x ) , the ELBO can be rewritten asL
Revised: ( z | x ) , the ELBO can be rewritten as

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: For explicit r α ( z | x ) , gradients of the ﬁrst term towards θ and φ can be easily computed, and KL can be estimated using KIVI.
Revised: For explicit r α ( z | x ) , gradients of the ﬁrst term towards θ and φ can be easily computed, and KL can be estimated using KIVI.

Original: Compared to prior, adaptive contrast estimates KL better if r α ( z | x ) approximates q well.
Revised: Compared to prior, adaptive contrast estimates KL better if r α ( z | x ) approximates q well.

Original: To make sure it is explicit, a commonly used adaptive distribution is Gaussian distribution whose mean µ ( x ) and standard derivation σ ( x ) matches with q .
Revised: To make sure it is explicit, a commonly used adaptive distribution is Gaussian distribution whose mean µ ( x ) and standard derivation σ ( x ) matches with q .

Original: According to KL divergence’s invariance under reparamterization, we have
Revised: According to KL divergence’s invariance under reparamterization, we have

Original: [Equation]
Revised: [Equation]

Original: Where ˆ q φ ( z | x ) denotes the distribution of ˆ z = z − µxσx and ˆ r 

Revised: Where ˆ q φ ( z | x ) denotes the distribution of ˆ z = z − µxσx and ˆ r

Original: (ˆ z ) denotes standard normal distribution.
Revised: (ˆ z ) denotes standard normal distribution.

Original: Under this reparameterization, we only need to estimate the density ratio between two distributions with 0 mean and 1 variance.
Revised: Under this reparameterization, we only need to estimate the density ratio between two distributions with 0 mean and 1 variance.

Original: In addition, the mean µ ( x ) and standard derivation σ ( x ) can be estimated with Monte Carlo from samples of q .
Revised: In addition, the mean µ ( x ) and standard derivation σ ( x ) can be estimated with Monte Carlo from samples of q .

Revised: E L OWER B OUND WITH G AMMA -P RIOR P RECISION

Revised: In the multivariate regression task, the output is sampled from a normal distribution with ˆ y ( x, W ) as mean and a parameter as variance.

Revised: The variance controls the likelihood of the model, therefore, choosing an appropriate variance is essential.

Revised: Therefore we place a Gamma prior Gamma (6 , 6) on its reciprocal (i.e., the precision of the Normal distribution).

Original: E A DDITIONAL E XPERIMENTS E.
Revised: The variational posterior we used is q ( W , λ ) = q ( W ) q ( λ ) , q ( λ ) ∼ Gamma ( α, β ) . Then the ELBO can be computed as

Original: [Figure]
Revised: [Equation]

Revised: where ψ ( x ) is the digamma function and KL ( q ( λ ) (cid:107) p ( λ )) can be calculated in closed-form.

Revised: F A DDITIONAL E XPERIMENT R ESULTS F. 2-D B AYESIAN L OGISTIC R EGRESSION

Original: We also conduct experiments on a 2-D Bayesian logistic regression example, which has an intractable posterior.
Revised: We also conduct experiments on a 2-D Bayesian logistic regression example, which has an intractable posterior.

Original: The model is
Revised: The model is

Original: [Equation]
Revised: [Equation]

Revised: [Figure]

Original: where w , x i ∈ R 2 ; σ is the sigmoid function.
Revised: where w , x i ∈ R 2 ; σ is the sigmoid function.

Original: N = 200 data points ( { ( x i , y i ) } 200 i =1 ) are generated from the true model as the training data (Fig. 5a).
Revised: N = 200 data points ( { ( x i , y i ) } 200 i =1 ) are generated from the true model as the training data (Fig. 5a).

Original: The unnormalized true posterior is plotted in
Revised: The unnormalized true posterior is plotted in

Original: Fig.
Revised: Fig.

Original: 5b.
Revised: 5b.

Original: As a baseline, we ﬁrst run VI with factorized normal distributions.
Revised: As a baseline, we ﬁrst run VI with factorized normal distributions.

Original: The result is shown in
Revised: The result is shown in

Original: Fig.
Revised: Fig.

Original: 5c.
Revised: 5c.

Original: It can be clearly seen that the factorized normal can capture the position and the scale of the true posterior but cannot ﬁt well to the shape due to its independence property across dimensions.
Revised: It can be clearly seen that the factorized normal can capture the position and the scale of the true posterior but cannot ﬁt well to the shape due to its independence property across dimensions.

Original: We then apply KIVI.
Revised: We then apply KIVI.

Original: The implicit posterior we use is a simple stochastic neural network (see
Revised: The implicit posterior we use is a simple stochastic neural network (see

Original: Appendix F.1).
Revised: Appendix G.1).

Original: To demonstrate how good the result is, we also run Hamiltonian Monte Carlo (HMC) to get posterior samples.
Revised: To demonstrate how good the result is, we also run Hamiltonian Monte Carlo (HMC) to get posterior samples.

Original: The results are plotted in Fig.
Revised: The results are plotted in Fig.

Original: 5d and 5e.
Revised: 5d and 5e.

Original: We can see that the implicit posterior is learned to capture the strong correlation between the two dimensions and can produce posterior samples that have a similar shape with samples drawn by HMC.
Revised: We can see that the implicit posterior is learned to capture the strong correlation between the two dimensions and can produce posterior samples that have a similar shape with samples drawn by HMC.

Original: E. C OMPARISON WITH N ORMALIZING F LOW
Revised: F. C OMPARISON WITH N ORMALIZING F LOW

Original: For comparing the results of KIVI with factorial variance inference and normalizing ﬂow, we use 100 samples, batch size 10 except for kin8nm and naval we use batch size 100.
Revised: For comparing the results of KIVI with factorial variance inference and normalizing ﬂow, we use 100 samples, batch size 10 except for kin8nm and naval we use batch size 100.

Original: We set the learning rate as 0.01 and run 500 epochs for 20 times.
Revised: We set the learning rate as 0.01 and run 500 epochs for 20 times.

Original: In NF, we apply 10 planar normalizing ﬂows on the weights to match the computation time of our implicit posterior.
Revised: In NF, we apply 10 planar normalizing ﬂows on the weights to match the computation time of our implicit posterior.

Original: The results are shown in Table 2.
Revised: The results are shown in Table 2.

Original: However, in the regression task normalizing ﬂows do not have improvements over VI with factorized normal posteriors.
Revised: However, in the regression task normalizing ﬂows do not have improvements over VI with factorized normal posteriors.

Original: We suspect that this is due to the limited form of normalizing ﬂows has led to optimization challenges, thus the inference procedure doesn’t take much beneﬁt from more ﬂexibility introduced by the ﬂow. 
Revised: We suspect that this is due to the limited form of normalizing ﬂows has led to optimization challenges, thus the inference procedure doesn’t take much beneﬁt from more ﬂexibility introduced by the ﬂow. and normalizing ﬂow, respectively.

Original: [Table]
Revised: [Table]

Original: F D ETAILS OF E XPERIMENTS F. T OY E XPERIMENTS
Revised: F. C HANGE OF I NTERPOLATION R ESULTS ON C ELEB A THROUGH TRAINING

Revised: We present here the generated images for the interpolation experiments on CelebA through the training process.

Revised: The images are generated after 1, 5, 10, 15, 20, 25 epochs.

Revised: The left side are results produced by AVB, and the right side for KIVI.

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Original:  1-D Gaussian Mixture We generate samples by forwarding a standard normal distribution through a two-layer MLP with 10 hidden units and one output unit.
Revised: [Figure] [Figure] G D ETAILS OF E XPERIMENTS G. T OY E XPERIMENTS 1-D Gaussian Mixture We generate samples by forwarding a standard normal distribution through a two-layer MLP with 10 hidden units and one output unit.

Original: We set regularization coefﬁcient λ as 0 .and density ratio clipping threshold 1e-8.
Revised: We set regularization coefﬁcient λ as 0 .and density ratio clipping threshold 1e-8.

Original: 2-D Bayesian Logistic Regression We compare factorized VI, KIVI and HMC in 2-D Bayesian logistic regression with N (0 , 1) prior.
Revised: 2-D Bayesian Logistic Regression We compare factorized VI, KIVI and HMC in 2-D Bayesian logistic regression with N (0 , 1) prior.

Original: The training data X are randomly sampled 200 points from
Revised: The training data X are randomly sampled 200 points from

Original: U[ − 5 , 5] × U[ − 5 , 5] , output Y are prediction of X with randomly sampled weights from prior.
Revised: U[ − 5 , 5] × U[ − 5 , 5] , output Y are prediction of X with randomly sampled weights from prior.

Original: For
Revised: For

Original: HMC, we run 100 chains, 200 iterations each, 10 leapfrog steps, and adaptive step size beginning from 1e-3.
Revised: HMC, we run 100 chains, 200 iterations each, 10 leapfrog steps, and adaptive step size beginning from 1e-3.

Original: We discard ﬁrst 100 samples generated.
Revised: We discard ﬁrst 100 samples generated.

Original: For factorized variational inference, we run for 100 epochs using full batch and 100 samples.
Revised: For factorized variational inference, we run for 100 epochs using full batch and 100 samples.

Original: Along the training, we anneal the learning rate linearly according to lr = [Figure] + [Figure]
Revised: Along the training, we anneal the learning rate linearly according to lr = 100 100+ epoch − 1 .

Original: − 1 .

Original: We follow the same setting as factorized variational inference for
Revised: We follow the same setting as factorized variational inference for

Original: KIVI, except that we use 1000 samples.
Revised: KIVI, except that we use 1000 samples.

Original: The regularization coefﬁcient λ and minimal density ratio interval are set to 0.1, 1e − 8 , respectively.
Revised: The regularization coefﬁcient λ and minimal density ratio interval are set to 0.1, 1e − 8 , respectively.

Original: For modelling the implicit variational posterior in KIVI, we forward 2-D random normal samples through 2 20-neuron fully connected layers getting h (we don’t use activation function for the last layer), then plus h with another random normal noise with adaptive variance getting z .
Revised: For modelling the implicit variational posterior in KIVI, we forward 2-D random normal samples through 2 20-neuron fully connected layers getting h (we don’t use activation function for the last layer), then plus h with another random normal noise with adaptive variance getting z .

Original: Finally we forward z through a 20-neuron layer and then 2-neuron linear layer getting the variational samples.
Revised: Finally we forward z through a 20-neuron layer and then 2-neuron linear layer getting the variational samples.

Original: F. V ARIATIONAL A UTOENCODERS
Revised: G. V ARIATIONAL A UTOENCODERS

Original: The decoders used in the MNIST experiment are MLPs with two hidden ReLU layers.
Revised: The decoders used in the MNIST experiment are MLPs with two hidden ReLU layers.

Original: The latent dimension is 8.
Revised: The latent dimension is 8.

Original: Each hidden layer is of size 500.
Revised: Each hidden layer is of size 500.

Original: The implicit posterior is also a MLP with two hidden
Revised: The implicit posterior is also a MLP with two hidden

Original: ReLU layers, with Gaussian noises of 500 dimension, zero mean and 500 dimensional adaptive variances added to the ﬁrst hidden layer.
Revised: ReLU layers, with Gaussian noises of 500 dimension, zero mean and 500 dimensional adaptive variances added to the ﬁrst hidden layer.

Original: The decoders used in the CelebA experiment are exactly the same structure with the one used for 64-size images in the DCGAN paper.
Revised: The decoders used in the CelebA experiment are exactly the same structure with the one used for 64-size images in the DCGAN paper.

Original: The latent dimension is 32.
Revised: The latent dimension is 32.

Original: The implicit posterior is a deep convolutional neural network with symmetric structure to the decoder, the output of the last convolutional layer is ﬂattened and is added a Gaussian noise of the same shape as the last dimension.
Revised: The implicit posterior is a deep convolutional neural network with symmetric structure to the decoder, the output of the last convolutional layer is ﬂattened and is added a Gaussian noise of the same shape as the last dimension.

Original: The noise is zero mean and has adaptive variances.
Revised: The noise is zero mean and has adaptive variances.

Original: Then the last hidden layer is of 500 hidden RelU units.
Revised: Then the last hidden layer is of 500 hidden RelU units.

Original: For AVB we use the same decoder.
Revised: For AVB we use the same decoder.

Revised: For both KIVI and AVB, we use batch size 64.

Revised: The other training parameters of AVB follow from its original code for CelebA.

Original: F. B AYESIAN N EURAL N ETWORKS F.3. R EGRESSION
Revised: G. B AYESIAN N EURAL N ETWORKS G.3. R EGRESSION

Original: As the regression datasets have small feature dimensions (usually less than 15, except 90 for year ), using Bayesian neural networks of one hidden layer (50 units) doesn’t produce very high-dimensional weights.
Revised: As the regression datasets have small feature dimensions (usually less than 15, except 90 for year ), using Bayesian neural networks of one hidden layer (50 units) doesn’t produce very high-dimensional weights.

Original: Therefore we still use fully connected neural networks for the variational posterior, where the samples of variational posterior is output of forwarding random normal distribution through a
Revised: Therefore we still use fully connected neural networks for the variational posterior, where the samples of variational posterior is output of forwarding random normal distribution through a

Original: MLP.
Revised: MLP.

Original: For all dataset, we use Relu as activation functions, we use MLP with one hidden layer except for yacht we use 2 hidden layers.
Revised: For all dataset, we use Relu as activation functions, we use MLP with one hidden layer except for yacht we use 2 hidden layers.

Original: We list the details in Table 3, which consists of 10 datasets and 2 weight matrices each.
Revised: We list the details in Table 3, which consists of 10 datasets and 2 weight matrices each.

Original: Take
Revised: Take

Original: Layer1 for boston as an example, (20, 30, N1) represents 20 random normal noise was generated and forwarded through a MLP with hidden neurons 30 and output neurons N1 = 14 × 50 , thus N corresponding to the number of weights in the ﬁrst layer of the Bayesian neural network.
Revised: Layer1 for boston as an example, (20, 30, N1) represents 20 random normal noise was generated and forwarded through a MLP with hidden neurons 30 and output neurons N1 = 14 × 50 , thus N corresponding to the number of weights in the ﬁrst layer of the Bayesian neural network.

Original: [Table]
Revised: [Table]

Revised: [Equation]

Original: F.3. C LASSIFICATION
Revised: G.3. C LASSIFICATION

Original: MNIST classiﬁcation needs a larger scale network than the one used in multivariate regression.
Revised: MNIST classiﬁcation needs a larger scale network than the one used in multivariate regression.

Original: Therefore we use a Matrix Multiplication Neural Network for modeling the variational posterior.
Revised: Therefore we use a Matrix Multiplication Neural Network for modeling the variational posterior.

Original: We experimented with hidden layer size 400, 800 and 1200, denoted by L. Below we set N = 500 when
Revised: We experimented with hidden layer size 400, 800 and 1200, denoted by L. Below we set N = 500 when

Original: L = 400 , otherwise N = 800 .
Revised: L = 400 , otherwise N = 800 .

Original: In the variational posterior network, we use two matrix multiplication hidden layers.
Revised: In the variational posterior network, we use two matrix multiplication hidden layers.

Original: For the ﬁrst layer parameter ( [785 , L ] ), the two hidden matrix multiplication layers are of size [ N, N ] and [ N, N ] with ReLU activations.
Revised: For the ﬁrst layer parameter ( [785 , L ] ), the two hidden matrix multiplication layers are of size [ N, N ] and [ N, N ] with ReLU activations.

Original: The ﬁnal output layer is of size [ L, 785] with linear activations.
Revised: The ﬁnal output layer is of size [ L, 785] with linear activations.

Original: The input matrix is random samples of size [30 , 30] from a standard normal distribution.
Revised: The input matrix is random samples of size [30 , 30] from a standard normal distribution.

Original: To be noted, each MM layer represents a left multiply, a right multiply and two sums.
Revised: To be noted, each MM layer represents a left multiply, a right multiply and two sums.

Original: For the second layer parameter ( [ L, L + 1] ), the two hidden matrix multiplication layers are of size [ N, N ] and [ N, N ] with ReLU activations.
Revised: For the second layer parameter ( [ L, L + 1] ), the two hidden matrix multiplication layers are of size [ N, N ] and [ N, N ] with ReLU activations.

Original: The ﬁnal output layer is of size [ L, L + 1] with linear activations.
Revised: The ﬁnal output layer is of size [ L, L + 1] with linear activations.

Original: The input matrix is random samples of size [30 , 30] from a standard normal distribution.
Revised: The input matrix is random samples of size [30 , 30] from a standard normal distribution.

Original: For the third layer parameter ( [10 , L + 1] ), the two hidden matrix multiplication layers are of size [30 , N ] and [30 , N ] with ReLU activations.
Revised: For the third layer parameter ( [10 , L + 1] ), the two hidden matrix multiplication layers are of size [30 , N ] and [30 , N ] with ReLU activations.

Original: The ﬁnal output layer is of size [10 , L + 1] with linear activations.
Revised: The ﬁnal output layer is of size [10 , L + 1] with linear activations.

Original: The input matrix is random samples of size [30 , 30] from a standard normal distribution.
Revised: The input matrix is random samples of size [30 , 30] from a standard normal distribution.

Original: With the variational posterior settings above, we perform experiments with KIVI and PC.
Revised: With the variational posterior settings above, we perform experiments with KIVI and PC.

Original: We use logistic regression as discriminator for PC.
Revised: We use logistic regression as discriminator for PC.

Original: We set regularization coefﬁcient λ as 0.001 and minimal density ratio threshold as 1e − 8 for KIVI.
Revised: We set regularization coefﬁcient λ as 0.001 and minimal density ratio threshold as 1e − 8 for KIVI.

Original: Both use 10 samples, batch size 100 and learning rate 0.
Revised: Both use 10 samples, batch size 100 and learning rate 0.

Original: for training.
Revised: for training.

Original: However, PC fails to converge because of the large scale latent space and KIVI shows promising results.
Revised: However, PC fails to converge because of the large scale latent space and KIVI shows promising results.

Original: G L OWER B OUND WITH G AMMA -P RIOR P RECISION

Original: In the multivariate regression task, the output is sampled from a normal distribution with ˆ y ( x, W ) as mean and a parameter as variance.

Original: The variance controls the likelihood of the model, therefore, choosing an appropriate variance is essential.

Original: Therefore we place a Gamma prior Gamma (6 , 6) on its reciprocal (i

Original: [Equation]

Original: e., the precision of the Normal distribution).

Original: The variational posterior we used is q ( W , λ )

Original: =

Original: q ( W ) q ( λ ) , q ( λ ) ∼ Gamma ( α, β ) .

Original: Then the ELBO can be computed as

Original: [Equation]

Original: where ψ ( x ) is the digamma function and KL ( q ( λ ) (cid:107) p ( λ )) can be calculated in closed-form

Original: [Equation]