Original: K ERNEL I MPLICIT V ARIATIONAL I NFERENCE
Revised: K ERNEL I MPLICIT V ARIATIONAL I NFERENCE

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Recent progress in variational inference has paid much attention to the ﬂexibility of variational posteriors.
Revised: Recent progress in variational inference has paid much attention to the ﬂexibility of variational posteriors.

Original: One promising direction is to use implicit distributions, i.e., distributions without tractable densities as the variational posterior.
Revised: One promising direction is to use implicit distributions, i.e., distributions without tractable densities as the variational posterior.

Original: However, existing methods on implicit posteriors still face challenges of noisy estimation and computational infeasibility when applied to models with high-dimensional latent variables.
Revised: However, existing methods on implicit posteriors still face challenges of noisy estimation and computational infeasibility when applied to models with high-dimensional latent variables.

Original: In this paper, we present a new approach named Kernel Implicit
Revised: In this paper, we present a new approach named Kernel Implicit

Original: Variational Inference that addresses these challenges.
Revised: Variational Inference that addresses these challenges.

Original: As far as we know, for the ﬁrst time implicit variational inference is successfully applied to Bayesian neural networks, which shows promising results on both regression and classiﬁcation tasks.
Revised: As far as we know, for the ﬁrst time implicit variational inference is successfully applied to Bayesian neural networks, which shows promising results on both regression and classiﬁcation tasks.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Bayesian methods have been playing vital roles in machine learning by providing a principled approach for generative modeling, posterior inference and preventing over-ﬁtting ( Ghahramani, 2015).
Revised: Bayesian methods have been playing vital roles in machine learning by providing a principled approach for generative modeling, posterior inference and preventing over-ﬁtting (Ghahramani, 2015).

Original: As it becomes a common practice to build deep models that have many parameters (LeCun et al., 2015), it is even more important to have a Bayesian formulation to capture the uncertainty in these models.
Revised: As it becomes a common practice to build deep models that have many parameters (LeCun et al., 2015), it is even more important to have a Bayesian formulation to capture the uncertainty in these models.

Original: For example, Bayesian Neural Networks (BNNs) (Neal, 2012; Blundell et al., 2015) have shown promise in reasoning about model conﬁdence and learning with few labeled data.
Revised: For example, Bayesian Neural Networks (BNNs) (Neal, 2012; Blundell et al., 2015) have shown promise in reasoning about model conﬁdence and learning with few labeled data.

Original: Another recent trend is to incorporate deep neural networks as a powerful function mapping between random variables in a Bayesian network, such as deep generative models like variational autoencoders (VAE) ( Kingma & Welling, 2013).
Revised: Another recent trend is to incorporate deep neural networks as a powerful function mapping between random variables in a Bayesian network, such as deep generative models like variational autoencoders (VAE) (Kingma & Welling, 2013).

Original: Except a few simple examples, Bayesian inference is typically challenging, for which variational inference (VI) has been a standard workhorse to approximate the true posterior ( Zhu et al., 2017).
Revised: Except a few simple examples, Bayesian inference is typically challenging, for which variational inference (VI) has been a standard workhorse to approximate the true posterior (Zhu et al., 2017).

Original: Traditional VI focuses on factorized variational posteriors to get analytical updates (known as Meanﬁeld VI).
Revised: Traditional VI focuses on factorized variational posteriors to get analytical updates (known as Meanﬁeld VI).

Original: While recent progress in this ﬁeld drives VI into stochastic, differentiable and amortized ( Hoffman et al., 2013; Paisley et al., 2012; Mnih & Gregor, 2014; Kingma & Welling, 2013), which does not rely on analytical updates anymore, factorized posteriors are still commonly used as the variational family.
Revised: While recent progress in this ﬁeld drives VI into stochastic, differentiable and amortized (Hoffman et al., 2013; Paisley et al., 2012; Mnih & Gregor, 2014; Kingma & Welling, 2013), which does not rely on analytical updates anymore, factorized posteriors are still commonly used as the variational family.

Original: This greatly restricts the ﬂexibility of the variational posterior, especially in high-dimensional spaces, which often leads to biased solutions as the true posterior is usually not factorized, thus not in the family.
Revised: This greatly restricts the ﬂexibility of the variational posterior, especially in highdimensional spaces, which often leads to biased solutions as the true posterior is usually not factorized, thus not in the family.

Original: There has been some works that try to improve the ﬂexibility of variational posteriors, borrowing ideas from invertible transformation of probability distributions (Rezende & Mohamed, 2015; Kingma et al., 2016).
Revised: There has been some works that try to improve the ﬂexibility of variational posteriors, borrowing ideas from invertible transformation of probability distributions (Rezende & Mohamed, 2015; Kingma et al., 2016).

Original: In their works, it is important for the transformation to be invertible to ensure that the transformed distribution has a tractable density.
Revised: In their works, it is important for the transformation to be invertible to ensure that the transformed distribution has a tractable density.

Original: Although utilizing invertible transformation is a promising direction to increase the expressiveness of the variational posterior, we argue that a more ﬂexible variational family can be constructed by using general deterministic or stochastic transformations, which are not necessarily invertible.
Revised: Although utilizing invertible transformation is a promising direction to increase the expressiveness of the variational posterior, we argue that a more ﬂexible variational family can be constructed by using general deterministic or stochastic transformations, which are not necessarily invertible.

Original: As a common result, the variational posterior we get in this way does not have a tractable density, despite that there is a way to sample from it.
Revised: As a common result, the variational posterior we get in this way does not have a tractable density, despite that there is a way to sample from it.

Original: This kind of distribution is called implicit distributions, and for variational methods that use an implicit variational posterior (also known as variational programs (Ranganath et al., 2016) or wild variational approximations (Liu & Feng, 2016)), we refer to them as Implicit Variational Inference (implicit VI).
Revised: This kind of distribution is called implicit distributions, and for variational methods that use an implicit variational posterior (also known as variational programs (Ranganath et al., 2016) or wild variational approximations (Liu & Feng, 2016)), we refer to them as Implicit Variational Inference (implicit VI).

Original: Most of the existing implicit VI methods (Mescheder et al., 2017; Husz´ar, 2017; Tran et al., 2017) rely on a discriminator to produce estimates of the variational objective and its gradients.
Revised: Most of the existing implicit VI methods (Mescheder et al., 2017; Husz´ar, 2017; Tran et al., 2017) rely on a discriminator to produce estimates of the variational objective and its gradients.

Original: As pointed out by many of them, the estimates are often noisy and can lead to unstable training.
Revised: As pointed out by many of them, the estimates are often noisy and can lead to unstable training.

Original: Besides, discriminator-based approaches are computationally infeasible when applied to nontrivial BNNs.
Revised: Besides, discriminator-based approaches are computationally infeasible when applied to nontrivial BNNs.

Original: In this paper we present an approach named Kernel Implicit Variational Inference (KIVI), which addresses the noisy estimation problem in previous works by providing a principled way of tuning the bias-variance tradeoff.
Revised: In this paper we present an approach named Kernel Implicit Variational Inference (KIVI), which addresses the noisy estimation problem in previous works by providing a principled way of tuning the bias-variance tradeoff.

Original: Furthermore, KIVI doesn’t rely on a discriminator and thus is computationally feasible for models with high-dimensional latent variables (e.g., BNNs).
Revised: Furthermore, KIVI doesn’t rely on a discriminator and thus is computationally feasible for models with high-dimensional latent variables (e.g., BNNs).

Original: KIVI is applicable to both global and local latent variable models, which is demonstrated by experiments on BNNs and VAEs.
Revised: KIVI is applicable to both global and local latent variable models, which is demonstrated by experiments on BNNs and VAEs.

Original: As far as we know, this is the ﬁrst time that implicit VI is successfully applied to BNNs, which shows promising results on both regression and classiﬁcation tasks.
Revised: As far as we know, this is the ﬁrst time that implicit VI is successfully applied to BNNs, which shows promising results on both regression and classiﬁcation tasks.

Original:  B ACKGROUND
Revised:  B ACKGROUND

Original: Consider a generative model p ( z ; x ) = p ( z ) p ( x j z ) , where x and z denote observed and latent variables, respectively.
Revised: Consider a generative model p ( z , x ) = p ( z ) p ( x | z ) , where x and z denote observed and latent variables, respectively.

Original: In VI, a variational distribution q ϕ ( z ) in some parametric family is chosen to approximate the true posterior p ( z j x ) by optimizing the evidence lower bound (ELBO):
Revised: In VI, a variational distribution q φ ( z ) in some parametric family is chosen to approximate the true posterior p ( z | x ) by optimizing the evidence lower bound (ELBO):

Original: [Equation]
Revised: [Equation]

Original: where KL denotes the Kullback-Leibler divergence KL( q ∥ p ) =
Revised: where KL denotes the Kullback-Leibler divergence KL( q (cid:107) p ) =

Original: E q [log qp ] .
Revised: E q [log qp ] .

Original: This objective is a lower bound of the log-likelihood log p ( x ) since it can be written as L ( x ; ϕ )
Revised: This objective is a lower bound of the log-likelihood log p ( x ) since it can be written as L ( x ; φ )

Original: = log p ( x ) (cid:0)
Revised: = log p ( x ) −

Original: KL( q ϕ ( z ) ∥ p ( z j x )) :
Revised: KL( q φ ( z ) (cid:107) p ( z | x )) .

Original: The maximum of this objective is achieved when q ϕ ( z ) =
Revised: The maximum of this objective is achieved when q φ ( z ) =

Original: p ( z j x ) .
Revised: p ( z | x ) .

Original: From Eq.
Revised: From Eq.

Original: (1), we can see that the challenge of using an implicit q ϕ is that computing KL( q ϕ ( z ) ∥ p ( z )) requires to evaluate the density of q ϕ , which is intractable for an implicit distribution.
Revised: (1), we can see that the challenge of using an implicit q φ is that computing KL( q φ ( z ) (cid:107) p ( z )) requires to evaluate the density of q φ , which is intractable for an implicit distribution.

Original: Recently, inspired by the probabilistic interpretation of Generative Adversarial Networks (GAN) (Goodfellow et al., 2014; Mohamed & Lakshminarayanan, 2016), there has been some works that extend the GAN approach to the posterior inference of latent variable models (LVMs) (Mescheder et al., 2017; Husz ´ ar, 2017; Tran et al., 2017).
Revised: Recently, inspired by the probabilistic interpretation of Generative Adversarial Networks (GAN) (Goodfellow et al., 2014; Mohamed & Lakshminarayanan, 2016), there has been some works that extend the GAN approach to the posterior inference of latent variable models (LVMs) (Mescheder et al., 2017; Husz´ar, 2017; Tran et al., 2017).

Original: These methods all use an implicit variational family and thus can be categorized into implicit VI methods.
Revised: These methods all use an implicit variational family and thus can be categorized into implicit VI methods.

Original: One of their key observations is that the density ratio q ϕ ( z ) p ( z ) can be estimated from samples of the two distributions by a probabilistic classiﬁer called the discriminator.
Revised: One of their key observations is that the density ratio q φ ( z ) p ( z ) can be estimated from samples of the two distributions by a probabilistic classiﬁer called the discriminator.

Original: They ﬁrst assign class labels ( y ) to q and p : Let samples from q ϕ ( z ) be of class y = 1 , and samples from p ( z ) be of class y = 0 .
Revised: They ﬁrst assign class labels ( y ) to q and p : Let samples from q φ ( z ) be of class y = 1 , and samples from p ( z ) be of class y = 0 .

Original: Given an equal class prior, the density ratio at a given point can be calculated as q ϕ ( z ) =p ( z ) =
Revised: Given an equal class prior, the density ratio at a given point can be calculated as q φ ( z ) /p

Original: p ( z j y = 1) =p ( z j y = 0)
Revised: ( z ) = p ( z | y = 1) /p

Original:  =
Revised: ( z | y = 0) =

Original: p ( y = 1 j z ) [-=p ( y = 
-]
Revised: p ( y = 1 | z ) [-=p ( y = 
-]/p

Original: j z ) ; which is the ratio between the class probabilities given the data point.
Revised: ( y = 0 | z ) , which is the ratio between the class probabilities given the data point.

Original: To estimate this, a discriminator D is trained to classify between the two classes, with a logistic loss:
Revised: To estimate this, a discriminator D is trained to classify between the two classes, with a logistic loss:

Original: [Equation]
Revised: [Equation]

Original: where D ( z ) outputs the probability of z ’s being from class y = 1 .
Revised: where D ( z ) outputs the probability of z ’s being from class y = 1 .

Original: Given that D is sufﬁciently ﬂexible, the optimal solution of Eq.
Revised: Given that D is sufﬁciently ﬂexible, the optimal solution of Eq.

Original: (2) is D ( z ) =
Revised: (2) is D ( z ) =

Original: q ϕ ( z ) = ( q ϕ ( z ) + p ( z )) 
Revised: q φ ( z ) / ( q φ ( z )+ p ( z )) .

Original: : Therefore, the KL divergence term in the ELBO of Eq.
Revised: Therefore, the KL divergence term in the ELBO of Eq.

Original: ( 1) can be approximated as KL( q ϕ ∥ p )(cid:25) E q ϕ ( z )
Revised: (1) can be approximated as KL( q φ (cid:107) p )

Original:  [log D ( z ) (cid:0) log(1 (cid:0) D ( z ))] : This is called prior-contrastive forms of VI in Husz ´ ar (2017).
Revised: ≈ E q φ ( z ) [log D ( z ) − log(1 − D ( z ))] . This is called prior-contrastive forms of VI in Husz ´ ar (2017).

Original: Note that the ratio approximation doesn’t change the gradients once the approximation is accurate ( Husz ´ ar, 2017).
Revised: Note that the ratio approximation doesn’t change the gradients once the approximation is accurate (Husz ´ ar, 2017).

Original: Though incorporating the discriminative power in a probabilistic model has shown great success in GANs, this method still suffers from challenging problems when applied to VI: (cid:15) Noisy density ratio estimation (DRE)
Revised: Though incorporating the discriminative power in a probabilistic model has shown great success in GANs, this method still suffers from challenging problems when applied to VI: • Noisy density ratio estimation (DRE)

Original: In VI, the variational posterior gets updated in each iteration.
Revised: In VI, the variational posterior gets updated in each iteration.

Original: As shown in Eq.
Revised: As shown in Eq.

Original: (2), the discriminator should be trained to optimum after each update.
Revised: (2), the discriminator should be trained to optimum after each update.

Original: However, in practice the inner loop for training the classiﬁer is often truncated to one or several iterations.
Revised: However, in practice the inner loop for training the classiﬁer is often truncated to one or several iterations.

Original: At the beginning of the inference procedure, it is hard for the classiﬁer to catch up with the variational posterior.
Revised: At the beginning of the inference procedure, it is hard for the classiﬁer to catch up with the variational posterior.

Original: The noisy signal produced by the discriminator leads to noisy gradients and thus unstable training.
Revised: The noisy signal produced by the discriminator leads to noisy gradients and thus unstable training.

Original: Besides, even if the discriminator quickly achieves the optimum in a small number of iterations, there is still another issue.
Revised: Besides, even if the discriminator quickly achieves the optimum in a small number of iterations, there is still another issue.

Original: Notice that the training loss in Eq.
Revised: Notice that the training loss in Eq.

Original: (2) is with expectations.
Revised: (2) is with expectations.

Original: But in practice we are using samples from the two distributions to approximate it.
Revised: But in practice we are using samples from the two distributions to approximate it.

Original: When the support of the distributions is high-dimensional, given the limited number of samples we use, the variance of this estimate is considerable, i.e., the discriminator tends to overﬁt the samples.
Revised: When the support of the distributions is high-dimensional, given the limited number of samples we use, the variance of this estimate is considerable, i.e., the discriminator tends to overﬁt the samples.

Original: The phenomenon is that the discriminator arrives at a state where samples are easily distinguished and the probabilities given by the discriminator are near 0 or 1, which is commonly observed in experiments (Mescheder et al., 2017).
Revised: The phenomenon is that the discriminator arrives at a state where samples are easily distinguished and the probabilities given by the discriminator are near 0 or 1, which is commonly observed in experiments (Mescheder et al., 2017).

Original: (cid:15) Computationally infeasible for high dimensional latent variables As the density ratio is estimated by a discriminator, the samples from the two distributions of latent variables should be fed into it.
Revised: • Computationally infeasible for high dimensional latent variables As the density ratio is estimated by a discriminator, the samples from the two distributions of latent variables should be fed into it.

Original: However, the typically used neural network discriminator cannot afford very highdimensional inputs (e.g., parameters in a moderate-size Bayesian neural network).
Revised: However, the typically used neural network discriminator cannot afford very high-dimensional inputs (e.g., parameters in a moderate-size Bayesian neural network).

Original:  K ERNEL I MPLICIT V ARIATIONAL I NFERENCE
Revised:  K ERNEL I MPLICIT V ARIATIONAL I NFERENCE

Original: To address the above challenges for implicit VI, we propose to replace the discriminator with a kernel method for DRE.
Revised: To address the above challenges for implicit VI, we propose to replace the discriminator with a kernel method for DRE.

Original: The advantages of this method are that it has a closed-form solution and that it allows us to explicitly tradeoff between bias and variance by tuning a regularization coefﬁcient.
Revised: The advantages of this method are that it has a closed-form solution and that it allows us to explicitly tradeoff between bias and variance by tuning a regularization coefﬁcient.

Original: 3. E STIMATING THE KL T ERM
Revised: 3. E STIMATING THE KL T ERM

Original: Speciﬁcally, let z 2 R d be the latent variable, and the true density ratio is r ( z )
Revised: Speciﬁcally, let z ∈ R d be the latent variable, and the true density ratio is r ( z )

Original: =
Revised: =

Original: q ( z ) = p ( z ) .
Revised: q ( z ) /p ( z ) .

Original: Consider modeling it with a function ^ r 2 H , where H is a Reproducing Kernel Hilbert Space (RKHS) induced by a positive deﬁnite kernel k ( z ; z ′ ) :
Revised: Consider modeling it with a function ˆ r ∈ H , where H is a Reproducing Kernel Hilbert Space (RKHS) induced by a positive deﬁnite kernel k ( z , z (cid:48) ) :

Original: R d (cid:2) R d ! R .
Revised: R d × R d → R .

Original: Similar to kernel ridge regression, we use an objective composed of a squared loss for regression plus a penalty for the complexity of the function.
Revised: Similar to kernel ridge regression, we use an objective composed of a squared loss for regression plus a penalty for the complexity of the function.

Original: For the squared loss we choose the form used by the unconstrained Least Square Importance Fitting (uLSIF) (Kanamori et al., 2009):
Revised: For the squared loss we choose the form used by the unconstrained Least Square Importance Fitting (uLSIF) (Kanamori et al., 2009):

Original: [Equation]
Revised: [Equation]

Original: where C is a constant, and approximate the expectation in J (^ r ) by Monte Carlo estimates:
Revised: where C is a constant, and approximate the expectation in J (ˆ r ) by Monte Carlo estimates:

Original: [Equation]
Revised: [Equation]

Original: where n p and n q are the number of samples from p and q , respectively.
Revised: where n p and n q are the number of samples from p and q , respectively.

Original: Note that in Eq.
Revised: Note that in Eq.

Original: ( 3), the base measure of the squared loss is chosen to be p so that the resulting form can be estimated without evaluating the density of both distributions.
Revised: (3), the base measure of the squared loss is chosen to be p so that the resulting form can be estimated without evaluating the density of both distributions.

Original: For the penalty term, the complexity of ^ r is measured by its RKHS norm ( ∥ ^ r ∥ 2 H ).
Revised: For the pernalty term, the complexity of ˆ r is measured by its RKHS norm ( (cid:107) ˆ r (cid:107) 2 H ).

Original: Putting them together, we get the ﬁnal objective:
Revised: Putting them together, we get the ﬁnal objective:

Original: [Equation]
Revised: [Equation]

Original: Here (cid:21) is the regularization coefﬁcient.
Revised: Here λ is the regularization coefﬁcient.

Original: Proposition 1.
Revised: Proposition 1.

Original: The optimal solution of Eq.
Revised: The optimal solution of Eq.

Original: (4) lies in the linear subspace spanned by the kernel functions centered at the samples ( f z pi g n p i =1 ; f z qj g n q j =1 ), i.e., ^ r has the form:
Revised: (4) lies in the linear subspace spanned by the kernel functions centered at the samples ( { z pi } n p i =1 , { z qj } n q j =1 ), i.e., ˆ r has the form:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: This can be seen as the generalization of the representer theorem (Sch ¨ olkopf et al., 2001) to the density ratio problem.
Revised: This can be seen as the generalization of the representer theorem (Sch ¨ olkopf et al., 2001) to the density ratio problem.

Original: So the proof follows the same procedure.
Revised: So the proof follows the same procedure.

Original: See Appendix A.
Revised: See Appendix A.

Original: Substitute Eq.
Revised: Substitute Eq.

Original: (5) into Eq.
Revised: (5) into Eq.

Original: (4) and let the derivatives w.r.t.
Revised: (4) and let the derivatives w.r.t.

Original: (cid:11) and (cid:12) be zeros, we get the optimal solution (a detailed derivation is given in Appendix B):
Revised: α and β be zeros, we get the optimal solution (a detailed derivation is given in Appendix B):

Original: [Equation]
Revised: [Equation]

Original: where ( K p ) i;j = 
Revised: where ( K p ) i,j = k

Original: k ( z pi ; z pj ) and ( K pq ) i;j = 
Revised: ( z pi , z pj ) and ( K pq ) i,j = k

Original: k ( z pi ; z qj ) .
Revised: ( z pi , z qj ) .

Original: We use the common RBF kernels k
Revised: We use the common RBF kernels k

Original: ( z ; z ′ ) =
Revised: ( z , z (cid:48) ) =

Original: exp( (cid:0)∥ z (cid:0)  z ′ ∥ 22 = 2 (cid:27) 2 ) .
Revised: exp (cid:0) −(cid:107) z − z (cid:48) (cid:107) 22 / 2 σ 2 (cid:1) .

Original: (cid:27) is the kernel bandwidth, which is determined by the commonly used median heuristic (the median of pairwise distances of the sample points).
Revised: σ is the kernel bandwidth, which is determined by the commonly used median heuristic (the median of pairwise distances of the sample points).

Original: Once we have the approximate density ratio function ^ r , a Monte Carlo estimate of KL( q ϕ ( z ) ∥ b ( z )) can be constructed by 1 n q ∑ n q i =1 log ^ r ( z qi ) .
Revised: Once we have the approximate density ratio function ˆ r , a Monte Carlo estimate of KL( q φ ( z ) (cid:107) p ( z )) can be constructed by 1 n q (cid:80) n q i =1 log ˆ r ( z q i ) .

Original: Note that there is a constraint that the estimated density ratio should be non-negative.
Revised: Note that there is a constraint that the estimated density ratio should be non-negative.

Original: However, we do not involve it in the optimization objective in order to get a closed-form solution, which indicates that some post-processing is needed to ensure this property.
Revised: However, we do not involve it in the optimization objective in order to get a closed-form solution, which indicates that some post-processing is needed to ensure this property.

Original: We solve the issue by clipping the estimated density ratio.
Revised: We solve the issue by clipping the estimated density ratio.

Original: The clipping values are searched from f 10 (cid:0) 8 ; 10 (cid:0) 16 ; 10 (cid:0) 32 g .
Revised: The clipping values are searched from { 10 − 8 , 10 − 16 , 10 − 32 } .

Original: In experiments we found that the algorithm is not sensitive to the clipping value.
Revised: In experiments we found that the algorithm is not sensitive to the clipping value.

Original: This is due to the accurate estimation guaranteed by the global optimum in the RKHS, which is a universal family when RBF kernels are used (Carmeli et al., 2010).
Revised: This is due to the accurate estimation guaranteed by the global optimum in the RKHS, which is a universal family when RBF kernels are used (Carmeli et al., 2010).

Original: The reverse ratio trick Another technique is essential to improve the estimation of the KL term, which we call the reverse ratio trick.
Revised: The reverse ratio trick Another technique is essential to improve the estimation of the KL term, which we call the reverse ratio trick.

Original: The key observation is that the  squared loss J (^ r ) in Eq.
Revised: The key observation is that the base measure of the squared loss J (ˆ r ) in Eq.

Original: (3) chooses p as the base measure, whereas the KL termto estimate ( KL( q ∥ p ) ) chooses q as the base measure.
Revised: (3) is p , whereas the base measure of the KL term ( KL( q (cid:107) p ) ) is q .

Original: Unless p and q match very well in where they put most probabilities, a small squared loss doesn’t always mean a good KL estimate.
Revised: Unless p and q match very well in where they put most probabilities, a small squared loss doesn’t always mean a good KL estimate.

Original: The solution is by a simple trick.
Revised: The solution is by a simple trick.

Original: Instead of estimating qp , we choose to estimate  pq and compute the KL term as (cid:0) E q log pq .
Revised: Instead of estimating qp , we choose to estimate r pq = pq and compute the KL term as − E q log pq .

Original: We denote the estimated reverse density ratio as ^ r pq , then the corresponding KL estimate is (cid:0) E q log ^ r pq . 
Revised: We denote the estimated reverse density ratio as ˆ r pq , then the corresponding KL estimate is − E q log ˆ r pq . Note that in this way the squared loss changes to J (ˆ r pq ) = 12 (cid:82) (ˆ r pq ( z ) − r pq

Original: In this way the ratio estimation is with the same base measure  as the KL term (both are with q ).
Revised: ( z )) 2 q ( z ) d z , which has the same base measure ( q ) as the KL term’s.

Original: As we shall see in experiments (Appendix F.1), this trick is essential to make the estimation sufﬁciently accurate for VI.
Revised: As we shall see in experiments (Appendix F.1), the trick is essential to make the estimation sufﬁciently accurate for VI.

Original: Gradient computation We now consider how to estimate the gradient of the KL term w.r.t.
Revised: Gradient computation We now consider how to estimate the gradient of the KL term w.r.t.

Original: variational parameters ϕ .
Revised: variational parameters φ .

Original: First it’s easy to prove as in Husz´ar (2017) that
Revised: First it’s easy to prove as in Husz´ar (2017) that

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: We provide the proof in Appendix C. Eq.
Revised: We provide the proof in Appendix C. Eq.

Original: (7) indicates that we can use any approximation of the density ratio, and the gradients w.r.t.
Revised: (7) indicates that we can use any approximation of the density ratio, and the gradients w.r.t.

Original: ϕ won’t change as long as the approximation is accurate.
Revised: φ won’t change as long as the approximation is accurate.

Original: We now replace the density ratio on the right side with ^ r pq : ∇ ϕ KL( q ϕ ∥ p ) (cid:25) (cid:0)∇ ϕ E q ϕ log ^ r pq : Then, the reparameterization trick (Kingma & Welling, 2013) can be used:
Revised: We now replace the density ratio on the right side with ˆ r pq : ∇ φ KL( q φ (cid:107) p ) ≈ −∇ φ E q φ log ˆ r pq . Then, the reparameterization trick (Kingma & Welling, 2013) can be used:

Original: [Equation]
Revised: [Equation]

Original: 3. T HE A LGORITHM
Revised: 3. T HE A LGORITHM

Original: We have constructed a closed-form estimate for the KL term and show its gradients can be estimated by the reparameterization trick.
Revised: We have constructed a closed-form estimate for the KL term and show its gradients can be estimated by the reparameterization trick.

Original: Note that the reparameterization trick can also be used to compute the gradients of the reconstruction term in Eq.
Revised: Note that the reparameterization trick can also be used to compute the gradients of the reconstruction term in Eq.

Original: (1) and thus can be applied to the ELBO.
Revised: (1) and thus can be applied to the ELBO.

Original: See Algo.for the complete algorithm.
Revised: See Algo. 1 for the complete algorithm.

Original: Note that the number of samples ( M ) used in the reconstruction term can be different from that required for the KL estimation, which can be reduced when the model is expensive (e.g., we set M = 1 in the experiments of VAEs).
Revised: Note that the number of samples ( M ) used in the reconstruction term can be different from that required for the KL estimation, which can be reduced when the model is expensive (e.g., we set M = 1 in the experiments of VAEs).

Original: Thus compared to normal reparameterized VI, the extra computational cost is mainly in calculating the inverse of the n p (cid:2) n p matrix in Eq.
Revised: Thus compared to normal reparameterized VI, the extra computational cost is mainly in calculating the inverse of the n p × n p matrix in Eq.

Original: As we shall see in experiments, tens or a hundred samples are sufﬁcient to obtain a stable KL estimate, so the added cost is not high. 
Revised: As we shall see in experiments, tens or a hundred samples are sufﬁcient to obtain a stable KL estimate, so the added cost is not high. Algorithm

Original: Require:
Revised: Require:

Original: Observed data x , model p (cid:18) ( x j z ) p ( z ) .
Revised: Observed data x , model p θ ( x | z ) p ( z ) .

Original: Require:
Revised: Require:

Original: Implicit variational posterior q ϕ ( z j x ) , n p , n q , M .
Revised: Implicit variational posterior q φ ( z | x ) , n p , n q , M .

Original: 1: repeat 2:
Revised: 1: repeat 2:

Original: Sample from prior: z pi (cid:24) p ( z ) ; i = 1 ; : : : ; n p .
Revised: Sample from prior: z pi ∼ p ( z ) , i = 1 , . . . , n p .

Original: Sample from variational: z qj (cid:24) q ( z j x ) ; j = 1 ; : : : ; n q .
Revised: Sample from variational: z qj ∼ q ( z | x ) , j = 1 , . . . , n q .

Original: Compute the density ratio ^ r pq by Eq.
Revised: Compute the density ratio ˆ r pq by Eq.

Original: (5), (6) and clip ^ r pq to be positive at z q s.
Revised: (5), (6) and clip ˆ r pq to be positive at z q s.

Original: Compute ^KL = (cid:0) 1 n q ∑ n q j =1 log ^ r pq ( z qj ) .
Revised: Compute ˆKL = − 1 n q (cid:80) n q j =1 log ˆ r pq ( z qj ) .

Original: Compute ^ L = 1 M ∑ Mm =1 log p ( x j z qm ) (cid:0) ^KL and estimate ∇ ϕ L with the reparameterization trick.
Revised: Compute ˆ L = 1 M (cid:80) Mm =1 log p ( x | z qm ) − ˆKL and estimate ∇ φ L with the reparameterization trick.

Original: Do gradient descent with ∇ ϕ L .
Revised: Do gradient descent with ∇ φ L .

Original: 8: (Optional)
Revised: 8: (Optional)

Original: For parameter learning, do gradient descent with ∇ (cid:18) L .
Revised: For parameter learning, do gradient descent with ∇ θ L .

Original: 9: until Convergence
Revised: 9: until Convergence

Original: KIVI addresses the two challenges stated in Sec. 2.
Revised: KIVI addresses the two challenges stated in Sec. 2.

Original: First, the ratio estimates are given in closedforms, thus not having the problem of not catching up.
Revised: First, the ratio estimates are given in closedforms, thus not having the problem of not catching up.

Original: Second, the bias-variance trade-off of the estimation can be controlled by the regularization coefﬁcient (cid:21) .
Revised: Second, the bias-variance trade-off of the estimation can be controlled by the regularization coefﬁcient λ .

Original: When (cid:21) is set smaller, the estimation is more aggressive to match the samples.
Revised: When λ is set smaller, the estimation is more aggressive to match the samples.

Original: When (cid:21) is set larger, the estimated ratio function is smoother.
Revised: When λ is set larger, the estimated ratio function is smoother.

Original: Choosing an appropriate (cid:21) , the variance of the gradients can be controlled, compared to the extreme ratio estimates given by classiﬁers when their output probabilities are near 0 or 1.
Revised: Choosing an appropriate λ , the variance of the gradients can be controlled, compared to the extreme ratio estimates given by classiﬁers when their output probabilities are near 0 or 1.

Original: Moreover, KIVI is directly applicable to both global and local latent variable models (LVMs), which is an advantage over nonparametric VI methods like particle mirror descent (Dai et al., 2015) and Stein variational gradient descent (SVGD) (Liu & Wang, 2016).
Revised: Moreover, KIVI is directly applicable to both global and local latent variable models (LVMs), which is an advantage over nonparametric VI methods like particle mirror descent (Dai et al., 2015) and Stein variational gradient descent (SVGD) (Liu & Wang, 2016).

Original: For the task of training local LVMs like VAEs, we additionally use the adaptive contrast (AC) technique (Mescheder et al., 2017), whose details are summarized in Appendix D.
Revised: For the task of training local LVMs like VAEs, we additionally use the adaptive contrast (AC) technique (Mescheder et al., 2017), whose details are summarized in Appendix D.

Original:  E XAMPLE : I MPLICIT V ARIATIONAL B AYESIAN N EURAL N ETWORKS
Revised:  E XAMPLE : I MPLICIT V ARIATIONAL B AYESIAN N EURAL N ETWORKS

Original: Now we present an example for using KIVI in Bayesian neural networks (BNNs), which have received increasing attention due to their ability to model uncertainty, an important factor in many tasks such as adversarial defense and reinforcement learning.
Revised: Now we present an example for using KIVI in Bayesian neural networks (BNNs), which have received increasing attention due to their ability to model uncertainty, an important factor in many tasks such as adversarial defense and reinforcement learning.

Original: However, despite that we have removed the need for a discriminator, it is still nontrivial to apply KIVI to BNNs because we need to design an implicit posterior that outputs very high-dimensional samples of latent variables (weights).
Revised: However, despite that we have removed the need for a discriminator, it is still nontrivial to apply KIVI to BNNs because we need to design an implicit posterior that outputs very high-dimensional samples of latent variables (weights).

Original: Existing implicit posteriors (Mescheder et al., 2017; Song et al., 2017) based on traditional fullyconnected neural networks cannot handle such a high-dimensional output space.
Revised: Existing implicit posteriors (Mescheder et al., 2017; Song et al., 2017) based on traditional fully-connected neural networks cannot handle such a high-dimensional output space.

Original: We present Matrix 
Revised: We present Matrix Multiplication

Original: Multiplication Neural Network (MMNN), an efﬁcient architecture for sampling large matrices.
Revised: Neural Network (MMNN), an efﬁcient architecture for sampling large matrices.

Original: Deploying MMNN, KIVI can easily scale up to large BNNs.
Revised: Deploying MMNN, KIVI can easily scale up to large BNNs.

Original: In BNNs, a prior is speciﬁed over the neural network parameters W = f W l g Ll =1 , where W l indicates weights in the l -th layer.
Revised: In BNNs, a prior is speciﬁed over the neural network parameters W = { W l } Ll =1 , where W l indicates weights in the l -th layer.

Original: Given an input x , the output y is modeled with
Revised: Given an input x , the output y is modeled with

Original: [Equation]
Revised: [Equation]

Original: where ^ y is the output of the feed-forward network f NN , and y is of a distribution P parameterized by ^ y and (cid:18) .
Revised: where ˆ y is the output of the feed-forward network f NN , and y is of a distribution P parameterized by ˆ y and θ .

Original: For regression, P is usually a Gaussian with ^ y as the mean.
Revised: For regression, P is usually a Gaussian with ˆ y as the mean.

Original: For classiﬁcation, P is usually a discrete distribution with ^ y as the unnormalized log probabilities.
Revised: For classiﬁcation, P is usually a discrete distribution with ˆ y as the unnormalized log probabilities.

Original: The true posterior of W in BNNs is intractable.
Revised: The true posterior of W in BNNs is intractable.

Original: Thus we turn to VI and use a variational posterior q to approximate it.
Revised: Thus we turn to VI and use a variational posterior q to approximate it.

Original: Denoting the dataset with X = f x i g N i =1 ; Y = 
Revised: Denoting the dataset with X = { x i } Ni =1 , Y = { y i } Ni =1 , we have the ELBO:

Original: f
Revised: [Equation]

Original: y
Revised: =

Original: i g N i =1 , we have the ELBO:
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Revised: −

Revised: [Equation]

Original:  The variational posterior is usually set to be factorized by layer: q ϕ ( W ) = 
Revised: [Equation] [Equation] (cid:107) p ( W )) . The variational posterior is usually set to be factorized by layer: q φ ( W ) = (cid:81)

Original: ∏ Ll =
Revised: Ll =

Original: 1 q ϕ l ( W l ) :
Revised: 1 q φ l ( W l ) .

Original: However, previous methods used variational posteriors with a limited capacity for each q ϕ l ( W l ) , including factorized Gaussian (Hernandez-Lobato & Adams, 2015), matrix variate Gaussian (Louizos & Welling, 2016; Sun et al., 2017) and normalizing ﬂows (Louizos & Welling, 2017).
Revised: However, previous methods used variational posteriors with a limited capacity for each q φ l ( W l ) , including factorized Gaussian (Hernandez-Lobato & Adams, 2015), matrix variate Gaussian (Louizos & Welling, 2016; Sun et al., 2017) and normalizing ﬂows (Louizos & Welling, 2017).

Original: Enabled to learn implicit variational posteriors, we propose to adopt a general distribution without an explicit density function, which has a form of
Revised: Enabled to learn implicit variational posteriors, we propose to adopt a general distribution without an explicit density function, which has a form of

Original: [Equation]
Revised: [Equation]

Original: Require:
Revised: Require:

Original: Input matrix X
Revised: Input matrix X

Original: Require: Network parameters f A li ; B li ; A ri ; B ri g Li = 1: for i = 1 ; : : : ; L do 2:
Revised: Require: Network parameters { A li , B li , A ri , B ri } Li = 1: for i = 1 , . . .

Revised: , L do 2:

Original: Left multiplication: X i =
Revised: Left multiplication: X i =

Original: A li X i (cid:0) 1 + B li 3:
Revised: A li X i − 1 + B li 3:

Original: Right multiplication: X i =
Revised: Right multiplication: X i =

Original: X i
Revised: X i

Original: A ri + B ri 4: if i (cid:20) L (cid:0) 1 then 5:
Revised: A ri + B ri 4: if i ≤ L − 1 then 5:

Original: X i =
Revised: X i =

Original: Relu ( X i ) 6: end if 7: end for 8: Output X L
Revised: Relu ( X i ) 6: end if 7: end for 8: Output X L

Original: Here g is a transformation parameterized by ϕ l , and W ql are treated as samples from q .
Revised: Here g is a transformation parameterized by φ l , and W ql are treated as samples from q .

Original: The key challenge is that W l are very high dimensional for moderate size neural networks.
Revised: The key challenge is that W l are very high dimensional for moderate size neural networks.

Original: Thus, we often cannot use a fully connected neural network (MLP) as g .
Revised: Thus, we often cannot use a fully connected neural network (MLP) as g .

Original: Inspired by low-rank matrix factorization ( Koren et al., 2009), we propose a new kind of network called Matrix Multiplication Neural Network (MMNN) to serve as g , as shown in Alg. 2.
Revised: Inspired by low-rank matrix factorization (Koren et al., 2009), we propose a new kind of network called Matrix Multiplication Neural Network (MMNN) to serve as g , as shown in Alg. 2.

Original: In each layer of an MMNN, an input matrix X i ( M in (cid:2) N in ) is left multiplied with a parameter matrix A li ( M out (cid:2) M in ) and is added a bias matrix B li ( M out (cid:2) N in ), then it is right multiplied with a parameter matrix A ri ( N in (cid:2) N out ) and is added a bias matrix B ri ( M out (cid:2) N out ).
Revised: In each layer of an MMNN, an input matrix X i ( M in × N in ) is left multiplied with a parameter matrix A li ( M out × M in ) and is added a bias matrix B li ( M out × N in ), then it is right multiplied with a parameter matrix A ri ( N in × N out ) and is added a bias matrix B ri ( M out × N out ).

Original: Finally, it is passed through a nonlinear activation (e.g., ReLU).
Revised: Finally it is passed through a nonlinear activation (e.g., ReLU).

Original: We call such a layer as an M out (cid:2) N out Matrix Multiplication layer.
Revised: We call such a layer as an M out × N out Matrix Multiplication layer.

Original: To model the implicit posterior of W l , we only need to randomly sample a matrix W 0 l of smaller size, and propagate it through the MMNN to get the output variational samples ( W ql ):
Revised: To model the implicit posterior of W l , we only need to randomly sample a matrix W 0 l of smaller size, and propagate it through the MMNN to get the output variational samples ( W ql ):

Original: [Equation]
Revised: [Equation]

Original: When modeling a matrix, MMNN has signiﬁcant computational advantages over MLPs, due to its low-rank property.
Revised: When modeling a matrix, MMNN has signiﬁcant computational advantages over MLPs, due to its low-rank property.

Original: For example, to model an M (cid:2) N weight matrix, consider a single-layer MMNN with the input matrix W 0 of size M 0 (cid:2) N 0 , the parameters needed are A l 1 ; B l 1 ; A r 1 ; B r 1 and they are in total of size MM 0 + MN 

Revised: For example, to model an M × N weight matrix, consider a single-layer MMNN with the input matrix W 0 of size M 0 × N 0 , the parameters needed are A l 1 , B l 1 , A r 1 , B r 1 and they are in total of size MM 0 + MN

Original: + N 0 N + MN , while if a single fully-connected layer is used, the parameter size is M 0 N 0 MN , which is much larger.
Revised: + N 0 N + MN , while if a single fully-connected layer is used, the parameter size is M 0 N 0 MN , which is much larger.

Original: Thus, we can use an MMNN as g in the variational posterior for normal-size neural networks.
Revised: Thus, we can use an MMNN as g in the variational posterior for normal-size neural networks.

Original: In tasks with very small networks, we still use an MLP as g .
Revised: In tasks with very small networks, we still use an MLP as g .

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Our work closely relates to the works on implicit generative models (IGMs, generative models that deﬁne implicit distributions) and density ratio estimation (DRE).
Revised: Our work closely relates to the works on implicit generative models (IGMs, generative models that deﬁne implicit distributions) and density ratio estimation (DRE).

Original: IGMs have drawn much attention due to the popularity of GANs.
Revised: IGMs have drawn much attention due to the popularity of GANs.

Original: General learning algorithms of implicit models have been surveyed in Mohamed & Lakshminarayanan (2016), where DRE plays a central role.
Revised: General learning algorithms of implicit models have been surveyed in Mohamed & Lakshminarayanan (2016), where DRE plays a central role.

Original: The connection between
Revised: The connection between

Original: DRE and GANs is also discussed in Uehara et al.
Revised: DRE and GANs is also discussed in Uehara et al.

Original: For a comprehensive review of DRE, we refer the readers to the survey (Hido et al., 2011).
Revised: For a comprehensive review of DRE, we refer the readers to the survey (Hido et al., 2011).

Original: We also refer the readers to many other works by
Revised: We also refer the readers to many other works by

Original: Sugiyama and his collaborators on DRE, such as KLIEP (Sugiyama et al., 2008), LSIF (Kanamori et al., 2009), and DRE based on Bregman divergence (Sugiyama et al., 2012).
Revised: Sugiyama and his collaborators on DRE, such as KLIEP (Sugiyama et al., 2008), LSIF (Kanamori et al., 2009), and DRE based on Bregman divergence (Sugiyama et al., 2012).

Original: Our work also builds upon the recent VI methods, including stochastic approximation by minibatches (Hoffman et al., 2013), direct gradient optimization of variational lower bounds (Paisley et al., 2012; Mnih & Gregor, 2014), and the reparameterization trick for training continuous LVMs (Kingma & Welling, 2013).
Revised: Our work also builds upon the recent VI methods, including stochastic approximation by mini-batches (Hoffman et al., 2013), direct gradient optimization of variational lower bounds (Paisley et al., 2012; Mnih & Gregor, 2014), and the reparameterization trick for training continuous LVMs (Kingma & Welling, 2013).

Original: Following the success of learning with IGMs, implicit distributions are applied to VI.
Revised: Following the success of learning with IGMs, implicit distributions are applied to VI.

Original: Many of them are based on discriminators, which can be divided into two categories: prior-contrastive (PC) and joint-contrastive (JC) (Husz´ar, 2017).
Revised: Many of them are based on discriminators, which can be divided into two categories: prior-contrastive (PC) and joint-contrastive (JC) (Husz´ar, 2017).

Original: In PC methods discriminators distinguish between samples from the prior and those from the variational posterior, while in JC methods they distinguish between the model joint distribution and the joint distribution composed of the data distribution and the variational posterior.
Revised: In PC methods discriminators distinguish between samples from the prior and those from the variational posterior, while in JC methods they distinguish between the model joint distribution and the joint distribution composed of the data distribution and the variational posterior.

Original: Concurrent with Husz´ar (2017), Mescheder et al.
Revised: Concurrent with Husz´ar (2017), Mescheder et al.

Original: (2017) proposed Adversarial Variational Bayes (AVB), which is an amortized version of PC methods for training local LVMs like VAEs.
Revised: (2017) proposed Adversarial Variational Bayes (AVB), which is an amortized version of PC methods for training local LVMs like VAEs.

Original: Prior to Husz´ar (2017), similar ideas with JC methods have been proposed in ALI ( Dumoulin et al., 2016) and Bi-GAN (Donahue et al., 2016).
Revised: Prior to Husz ´ ar (2017), similar ideas with JC methods have been proposed in ALI (Dumoulin et al., 2016) and Bi-GAN (Donahue et al., 2016).

Original: Nonparametric VI methods such as PMD (Dai et al., 2015) and SVGD (Liu & Wang, 2016) that adapt a set of particles towards the true posterior are also closely related to implicit VI.
Revised: Nonparametric VI methods such as PMD (Dai et al., 2015) and SVGD (Liu & Wang, 2016) that adapt a set of particles towards the true posterior are also closely related to implicit VI.

Original: They share the similar advantage of ﬂexible approximations.
Revised: They share the similar advantage of ﬂexible approximations.

Original: More recently, the amortized version of SVGD has been developed ( Liu & Feng, 2016) and the same idea has been applied to MCMC (Li et al., 2017).
Revised: More recently, the amortized version of SVGD has been developed (Liu & Feng, 2016) and the same idea has been applied to MCMC (Li et al., 2017).

Original: It was further shown in Li & Turner (2017) that the core identity in SVGD (Stein’s identity) could also be employed to approximate the gradients of implicit distributions.
Revised: It was further shown in Li & Turner (2017) that the core identity in SVGD (Stein’s identity) could also be employed to approximate the gradients of implicit distributions.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We present empirical results on both synthetic and real datasets to demonstrate the beneﬁts of KIVI.
Revised: We present empirical results on both synthetic and real datasets to demonstrate the beneﬁts of KIVI.

Revised: 6. T OY 1-D G AUSSIAN M IXTURES

Original: [Figure]
Revised: [Figure]

Original: We ﬁrstly conduct a toy experiment to approximate a 1-D Gaussian mixture distribution with VI.
Revised: We ﬁrstly conduct a toy experiment to approximate a 1-D Gaussian mixture distribution with VI.

Original: The Gaussian mixture distribution has two equally distributed unit-variance components whose means are -3 and 3.
Revised: The Gaussian mixture distribution has two equally distributed unit-variance components whose means are -3 and 3.

Original: We compare
Revised: We compare

Original: KIVI with VI using a single Gaussian posterior (Fig. 1).
Revised: KIVI with VI using a single Gaussian posterior (Fig. 1).

Original: The variational distribution used by
Revised: The variational distribution used by

Original: KIVI generates samples by propagating a standard normal distribution through a two-layer
Revised: KIVI generates samples by propagating a standard normal distribution through a two-layer

Original: MLP with 10 hidden units in each layer and one output unit.
Revised: MLP with 10 hidden units in each layer and one output unit.

Original: As shown, the Gaussian posterior converges to a single mode.
Revised: As shown, the Gaussian posterior converges to a single mode.

Original: In contrast, KIVI can accurately approximate the two modes with an expressive variational posterior.
Revised: In contrast, KIVI can accurately approximate the two modes with an expressive variational posterior.

Original: We defer another toy experiment on 2-D Bayesian logistic regression to Appendix F.1, where the importance of the reverse ratio trick is illustrated.
Revised: We defer another toy experiment on 2-D Bayesian logistic regression to Appendix F.1, where the importance of the reverse ratio trick is illustrated.

Original: 6. B AYESIAN N EURAL N ETWORKS (BNN S )
Revised: 6. B AYESIAN N EURAL N ETWORKS (BNN S )

Original: As stated in Sec. 4, the latent variables in BNNs are global to all data points and are usually very high-dimensional, for which a ﬂexible variational family is essential.
Revised: As stated in Sec. 4, the latent variables in BNNs are global to all data points and are usually very high-dimensional, for which a ﬂexible variational family is essential.

Original: We compare KIVI with stateof-the-art VI methods by doing regression and classiﬁcation on standard benchmarks.
Revised: We compare KIVI with state-of-the-art VI methods by doing regression and classiﬁcation on standard benchmarks.

Original: 6.2. R EGRESSION
Revised: 6.2. R EGRESSION

Original: To quantitatively measure the predictive ability of BNNs with KIVI as the inference method, we use standard multivariate regression benchmarks from recent works (Table 1), such as probabilistic backpropagation (PBP) (Hernandez-Lobato & Adams, 2015).
Revised: To quantitatively measure the predictive ability of BNNs with KIVI as the inference method, we use standard multivariate regression benchmarks from recent works (Table 1), such as probabilistic backpropagation (PBP) (Hernandez-Lobato & Adams, 2015).

Original: We compare with state-of-the-art
Revised: We compare with state-of-the-art

Original: [Table]
Revised: [Table]

Original: methods: the Bayesian interpretation of dropout (Gal & Ghahramani, 2016) and stein variational gradient descent (SVGD) (Liu & Wang, 2016) 1 .
Revised: methods: the Bayesian interpretation of dropout (Gal & Ghahramani, 2016) and stein variational gradient descent (SVGD) (Liu & Wang, 2016) 1 .

Original: Following the setup in PBP, we use BNNs with one 50-unit hidden layer except in the two large datasets, i.e., Protein Structure and Year Predication
Revised: Following the setup in PBP, we use BNNs with one 50-unit hidden layer except in the two large datasets, i.e., Protein Structure and Year Predication

Original: MSD , where 100 units are used.
Revised: MSD , where 100 units are used.

Original: We randomly select 90% of the whole dataset for training and leave the rest for testing.
Revised: We randomly select 90% of the whole dataset for training and leave the rest for testing.

Original: We also put a Gamma prior on the precision of the observation noise to adaptively learn it (see Appendix E).
Revised: We also put a Gamma prior on the precision of the observation noise to adaptively learn it (see Appendix E).

Original: For all datasets, we set n p = n q = M = 100 ; (cid:21) = 0 : 001 and set the batch size as 100 and the learning rate as 0.001.
Revised: For all datasets, we set n p = n q = M = 100 , λ = 0 . 001 and set the batch size as 100 and the learning rate as 0.001.

Original: The model is trained for 3000 epochs for the small datasets with less than 1000 data points, and 500 epochs for the others.
Revised: The model is trained for 3000 epochs for the small datasets with less than 1000 data points, and 500 epochs for the others.

Original: We report the mean errors and standard deviations averaged over 20 runs, except 5 runs for Protein Structure and 1 run for 
Revised: We report the mean errors and standard deviations averaged over 20 runs, except 5 runs for Protein Structure and 1 run for Year

Original: Year Predication MSD .
Revised: Predication MSD .

Original: As networks used in these tasks are of a small scale, we use MLPs with one hidden layer in the implicit variational posterior (see Appendix G.2.1 for details).
Revised: As networks used in these tasks are of a small scale, we use MLPs with one hidden layer in the implicit variational posterior (see Appendix G.2.1 for details).

Original: Table 1 shows the results with the best ones marked in bold.
Revised: Table 1 shows the results with the best ones marked in bold.

Original: Results of SVGD and dropout are cited from their papers, which have the same setting as ours.
Revised: Results of SVGD and dropout are cited from their papers, which have the same setting as ours.

Original: We can see that KIVI consistently outperforms SVGD and dropout on both RMSE and test-LL for most datasets.
Revised: We can see that KIVI consistently outperforms SVGD and dropout on both RMSE and test-LL for most datasets.

Original: Especially on RMSE,
Revised: Especially on RMSE,

Original: KIVI has signiﬁcant improvements over them except on Wine and Year Predication MSD .
Revised: KIVI has signiﬁcant improvements over them except on Wine and Year Predication MSD .

Original: It suggests that KIVI enables the implicit variational posterior to capture the predictive uncertainty in network parameters, which is hard to be fully described by a mixture of two delta distributions (dropout) and a ﬁxed set of particles (SVGD).
Revised: It suggests that KIVI enables the implicit variational posterior to capture the predictive uncertainty in network parameters, which is hard to be fully described by a mixture of two delta distributions (dropout) and a ﬁxed set of particles (SVGD).

Original: We emphasize that although the nonparametric nature of SVGD has also made the approximation more ﬂexible, it uses the same set of particles throughout the inference procedure, while each iteration of KIVI generates a new set of particles.
Revised: We emphasize that although the nonparametric nature of SVGD has also made the approximation more ﬂexible, it uses the same set of particles throughout the inference procedure, while each iteration of KIVI generates a new set of particles.

Original: Thus the implicit posterior learned by KIVI is smoothed by the parametric model.
Revised: Thus the implicit posterior learned by KIVI is smoothed by the parametric model.

Original: Recently, normalizing ﬂows have shown good performance on BNNs ( Louizos & Welling, 2017).
Revised: Recently, normalizing ﬂows have shown good performance on BNNs (Louizos & Welling, 2017).

Original: So we also experiment with directly applying normalizing ﬂows to this task.
Revised: So we also experiment with directly applying normalizing ﬂows to this task.

Original: The results are reported in Appendix F.2.
Revised: The results are reported in Appendix F.2.

Original: 6.2. C LASSIFICATION
Revised: 6.2. C LASSIFICATION

Original: For classiﬁcation, we present the results on MNIST, which consists of 60,000 training images and 10,000 test images of handwriting digits.
Revised: For classiﬁcation, we present the results on MNIST, which consists of 60,000 training images and 10,000 test images of handwriting digits.

Original: Compared to the above datasets in regression, MNIST has a much higher feature dimension, introducing millions of parameters even in moderate-size networks, which brings big challenges for BNNs.
Revised: Compared to the above datasets in regression, MNIST has a much higher feature dimension, introducing millions of parameters even in moderate-size networks, which brings big challenges for BNNs.

Original: As a standard benchmark, the performance on MNIST can be improved by many other techniques, such as convolution, generative pre-training, data augmentation, etc.
Revised: As a standard benchmark, the performance on MNIST can be improved by many other techniques, such as convolution, generative pre-training, data augmentation, etc.

Original: To ensure a fair comparison, we follow the settings from Bayes-By-Backprop (Blundell et al., 2015) and focus on improving the performance of ordinary MLPs without using any of these techniques.
Revised: To ensure a fair comparison, we follow the settings from Bayes-By-Backprop (Blundell et al., 2015) and focus on improving the performance of ordinary MLPs without using any of these techniques.

Original: The network structures used are three MLPs, all with two ReLU hidden layers, and the layer sizes are 400, 800 and 1200, respectively.
Revised: The network structures used are three MLPs, all with two ReLU hidden layers, and the layer sizes are 400, 800 and 1200, respectively.

Original: For KIVI, we used MMNNs with two hidden matrix multiplication layers in the implicit posterior (see Appendix G.2.2 for details).
Revised: For KIVI, we used MMNNs with two hidden matrix multiplication layers in the implicit posterior (see Appendix G.2.2 for details).

Original: We set n p = n q = M = 10 ; (cid:21) = 0 : 001 , batch size 100 and train for 300 epochs.
Revised: We set n p = n q = M = 10 , λ = 0 . 001 , batch size 100 and train for 300 epochs.

Original: The initial learning rate is 0.001 and is annealed every 100 epochs by ratio 0.5.
Revised: The initial learning rate is 0.001 and is annealed every 100 epochs by ratio 0.5.

Original: We used the last 10,000 samples of the training set as the validation set for model selection. 
Revised: We used the last 10,000 samples of the training set as the validation set for model selection. ours.

Original: [Figure]
Revised: [Figure]

Original: Fig.
Revised: Fig.

Original: 2 summarizes the results.
Revised: 2 summarizes the results.

Original: We can see that KIVI achieves better accuracy compared to plain SGD ( Simard et al., 2003), dropout (Srivastava et al., 2014) and Bayes-By-Backprop (Blundell et al., 2015) on all three types of MLPs.
Revised: We can see that KIVI achieves better accuracy compared to plain SGD (Simard et al., 2003), dropout (Srivastava et al., 2014) and Bayes-By-Backprop (Blundell et al., 2015) on all three types of MLPs.

Original: KIVI even performs better than Bayes-By-Backprop with a changed prior (scale mixture), which makes the model itself more ﬂexible than ours.
Revised: KIVI even performs better than Bayes-By-Backprop with a changed prior (scale mixture), which makes the model itself more ﬂexible than ours.

Original: When the layer size is 800, our result is comparable to an ensemble of 5 networks with dropconnect (Wan et al., 2013), which demonstrates that the implicit posterior has been learned to account for most model uncertainty.
Revised: When the layer size is 800, our result is comparable to an ensemble of 5 networks with dropconnect (Wan et al., 2013), which demonstrates that the implicit posterior has been learned to account for most model uncertainty.

Original: We also conduct experiments with the prior-contrastive (PC) method (the counterpart of AVB for global latent variables, see Sec. 2).
Revised: We also conduct experiments with the prior-contrastive (PC) method (the counterpart of AVB for global latent variables, see Sec. 2).

Original: The key challenge for applying PC here is that posterior samples are extremely high-dimensional, and if fed into discriminators like neural networks, they will cause unaffordable computation cost.
Revised: The key challenge for applying PC here is that posterior samples are extremely high-dimensional, and if fed into discriminators like neural networks, they will cause unaffordable computation cost.

Original: To get around this, we use a logistic regression as the discriminator in PC.
Revised: To get around this, we use a logistic regression as the discriminator in PC.

Original: The experiment settings of PC are reported in Appendix G.2.2.
Revised: The experiment settings of PC are reported in Appendix G.2.2.

Original: The training lower bounds of the two methods are plotted in Fig.
Revised: The training lower bounds of the two methods are plotted in Fig.

Original: We can see that in the beginning they increase at the same pace, then PC fails to converge with lower bound explosion while KIVI improves consistently.
Revised: We can see that in the beginning they increase at the same pace, then PC fails to converge with lower bound explosion while KIVI improves consistently.

Original: The explosion is mainly because the input to the discriminator is of hundreds of thousands of dimensions, and plain logistic regression cannot produce reliable density ratio estimates.
Revised: The explosion is mainly because the input to the discriminator is of hundreds of thousands of dimensions, and plain logistic regression cannot produce reliable density ratio estimates.

Original: We also experiment with PC for layer size 800 and 1200.
Revised: We also experiment with PC for layer size 800 and 1200.

Original: They both fail to converge in the end.
Revised: They both fail to converge in the end.

Original: 6. V ARIATIONAL A UTOENCODERS
Revised: 6. V ARIATIONAL A UTOENCODERS

Original: [Figure]
Revised: [Figure]

Original: As stated in Sec. 3.2, KIVI is applicable to both global and local LVMs, and the latter can be learned using an amortized scheme (i.e., use q ϕ ( z j x ) instead of q ϕ ( z ) ).
Revised: As stated in Sec. 3.2, KIVI is applicable to both global and local LVMs, and the latter can be learned using an amortized scheme (i.e., use q φ ( z | x ) instead of q φ ( z ) ).

Original: Here we present an application on training variational autoencoders (VAE) with implicit posteriors to demonstrate this.
Revised: Here we present an application on training variational autoencoders (VAE) with implicit posteriors to demonstrate this.

Original: The generative process of VAEs proceeds as
Revised: The generative process of VAEs proceeds as

Original: [Equation]
Revised: [Equation]

Original: where z are latent features, x are observations, and P is the output distribution for modeling the observations, which takes the outputs of the neural network ( f NN ( z ) ) as parameters.
Revised: where z are latent features, x are observations, and P is the output distribution for modeling the observations, which takes the outputs of the neural network ( f NN ( z ) ) as parameters.

Original: We conduct experiments on two widely used datasets for generative modeling: binarized MNIST and CelebA (Liu et al., 2015).
Revised: We conduct experiments on two widely used datasets for generative modeling: binarized MNIST and CelebA (Liu et al., 2015).

Original: P is a Bernoulli distribution for binarized MNIST, while a normal distribution for
Revised: P is a Bernoulli distribution for binarized MNIST, while a normal distribution for

Original: CelebA. For VAE the variational lower bound has the same form as Eq.
Revised: CelebA. For VAE the variational lower bound has the same form as Eq.

Original: (1), except that q ϕ ( z ) is replaced by q ϕ ( z j x ) .
Revised: (1), except that q φ ( z ) is replaced by q φ ( z | x ) .

Original: The original VAE parameterizes q ϕ ( z j x ) as
Revised: The original VAE parameterizes q φ ( z | x ) as

Original: [Equation]
Revised: [Equation]

Original: where g NN is a neural network that outputs the parameters of the normal distribution.
Revised: where g NN is a neural network that outputs the parameters of the normal distribution.

Original: In the MNIST case, g NN is an MLP with two hidden layers, which is illustrated in Fig.
Revised: In the MNIST case, g NN is an MLP with two hidden layers, which is illustrated in Fig.

Original: 3a (left).
Revised: 3a (left).

Original: To form an implicit posterior, a direct choice is to move the stochastic noise from the output layer to the penultimate hidden layer, as illustrated by Fig.
Revised: To form an implicit posterior, a direct choice is to move the stochastic noise from the output layer to the penultimate hidden layer, as illustrated by Fig.

Original: 3a (right).
Revised: 3a (right).

Original: Before applying KIVI, a crucial question is what we expect to get by using implicit posteriors for training VAEs.
Revised: Before applying KIVI, a crucial question is what we expect to get by using implicit posteriors for training VAEs.

Original: One target could be that we may get tighter lower bounds of the data log-likelihood (LL) because the algorithm searches in a larger variational family for the optimal lower bound.
Revised: One target could be that we may get tighter lower bounds of the data log-likelihood (LL) because the algorithm searches in a larger variational family for the optimal lower bound.

Original: This suggests, however in a very weak way, that doing optimization in the larger space will lead to better test LL, given the optimization always arrives at local optima.
Revised: This suggests, however in a very weak way, that doing optimization in the larger space will lead to better test LL, given the optimization always arrives at local optima.

Original: Previously Adversarial Variational
Revised: Previously Adversarial Variational

Original: Bayes (AVB) has shown some results on MNIST, by comparing the test LL of models trained by plain VAE and AVB, using golden truths estimated by annealed importance sampling (AIS)
Revised: Bayes (AVB) has shown some results on MNIST, by comparing the test LL of models trained by plain VAE and AVB, using golden truths estimated by annealed importance sampling (AIS)

Original: (Wu et al. , 2016).
Revised: (Wu et al., 2016).

Original: However, for the results reported in AVB, the model architectures used by plain VAE and AVB are very different, which leads to concerns about which part of the change contributes to the improved likelihoods.
Revised: However, for the results reported in AVB, the model architectures used by plain VAE and AVB are very different, which leads to concerns about which part of the change contributes to the improved likelihoods.

Original: Here we adopt another setting to better demonstrate the gain from implicit posteriors.
Revised: Here we adopt another setting to better demonstrate the gain from implicit posteriors.

Original: We observe that the key improvement of implicit posteriors is that objectives with them average over a much broader range of posterior conﬁgurations.
Revised: We observe that the key improvement of implicit posteriors is that objectives with them average over a much broader range of posterior conﬁgurations.

Original: This effect not only contributes to a larger search space that contains tighter lower bound values, but also makes the VAE model better prevent overﬁtting.
Revised: This effect not only contributes to a larger search space that contains tighter lower bound values, but also makes the VAE model better prevent overﬁtting.

Original: To verify that KIVI keeps this property, we conduct experiments on a statically binarized MNIST dataset and use models with no prior knowledge of the problem (MLP instead of convnets), which is a typical setting that leads to overﬁtting.
Revised: To verify that KIVI keeps this property, we conduct experiments on a statically binarized MNIST dataset and use models with no prior knowledge of the problem (MLP instead of convnets), which is a typical setting that leads to overﬁtting.

Original: The latent dimension of the VAE model is 8, and f NN ( z ) is an MLP with two hidden layers of size 500.
Revised: The latent dimension of the VAE model is 8, and f NN ( z ) is an MLP with two hidden layers of size 500.

Original: The parameters for KIVI are n p = n q = 100 ; M = 1 .
Revised: The parameters for KIVI are n p = n q = 100 , M = 1 .

Original: More details can be found in Appendix G.3.
Revised: More details can be found in Appendix G.3.

Original: Fig 3b shows the training and testing curves of VAEs with or without KIVI.
Revised: Fig 3b shows the training and testing curves of VAEs with or without KIVI.

Original: It can be seen that the lower bound gap between the training and testing curves of the plain VAE are much larger than that of KIVI, which indicates that the VAE trained by KIVI is less prone to overﬁtting.
Revised: It can be seen that the lower bound gap between the training and testing curves of the plain VAE are much larger than that of KIVI, which indicates that the VAE trained by KIVI is less prone to overﬁtting.

Original: After 3k epochs we evaluate the test LL on 2048 test images using AIS and get -97.3 for the plain VAE, while -94.8 for the VAE trained by KIVI.
Revised: After 3k epochs we evaluate the test LL on 2048 test images using AIS and get -97.3 for the plain VAE, while -94.8 for the VAE trained by KIVI.

Original: To demonstrate that KIVI scales to larger models, we trained a VAE with the same network structure used by DCGAN (Radford et al., 2015) on CelebA using KIVI.
Revised: To demonstrate that KIVI scales to larger models, we trained a VAE with the same network structure used by DCGAN (Radford et al., 2015) on CelebA using KIVI.

Original: The latent dimension in this case is 32.
Revised: The latent dimension in this case is 32.

Original: The implicit posterior is constructed in a way similar to the one shown in Fig.
Revised: The implicit posterior is constructed in a way similar to the one shown in Fig.

Original: 3a, with the bottom hidden layers symmetric to the decoder network (See Appendix G.3 for details).
Revised: 3a, with the bottom hidden layers symmetric to the decoder network (See Appendix G.3 for details).

Original: To visually check the latent space learned by KIVI, we show the  between the latent z vector of two real images after 25 epochs (Fig. 4), when the model has converged.
Revised: To visually check the latent space learned by KIVI, we show the reconstruction results of linearly interpolating between the latent z vector of two real images after 25 epochs (Fig. 4)

Original: Compared to the latent space learned by AVB after the same epochs (we use the public code from AVB and set the same decoder structure), we ﬁnd ours are smoother, and the interpolated images are much sharper.
Revised: Compared to the latent space learned by AVB after the same epochs (we use the public code from AVB and set the same decoder structure), we ﬁnd ours are smoother, and the interpolated images are much sharper.

Original: More interpolation results through the training process are presented and compared in Appendix F.5.
Revised: More interpolation results through the training process are presented and compared in Appendix F.5.

Original: We also use the learned model to generate 10,000 images and evaluate the sample quality using Fr´echet Inception Distance (FID) (Heusel et al., 2017).
Revised: We also use the learned model to generate 10,000 images and evaluate the sample quality using Fr´echet Inception Distance (FID) (Heusel et al., 2017).

Original: The FID score achieved at epoch 25 by AVB and KIVI are 160 and 41, respectively (smaller is better).
Revised: The FID score achieved at epoch 25 by AVB and KIVI are 160 and 41, respectively (smaller is better).

Original: In fact many efforts are required to make AVB successfully train a model for CelebA to produce results shown in the ﬁgure.
Revised: In fact many efforts are required to make AVB successfully train a model for CelebA to produce results shown in the ﬁgure.

Original: As reported in Mescheder et al.
Revised: As reported in Mescheder et al.

Original: (2017), the log prior log p ( z ) is explicitly added to the discriminator ( T ( x ; z ) ), while KIVI doesn’t need much tuning: there is no need to carefully design a discriminator, and the only two hyper-parameters (i.e., (cid:21) and the clipping value) both have clear meanings.
Revised: (2017), the log prior log p ( z ) is explicitly added to the discriminator ( T ( x , z ) ), while KIVI doesn’t need much tuning: there is no need to carefully design a discriminator, and the only two hyper-parameters (i.e., λ and the clipping value) both have clear meanings.

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: We present an implicit VI method named Kernel Implicit Variational Inference (KIVI), which provides a principled way of tuning bias-variance tradeoff and makes implicit VI computationally feasible for models with high-dimensional latent variables.
Revised: We present an implicit VI method named Kernel Implicit Variational Inference (KIVI), which provides a principled way of tuning bias-variance tradeoff and makes implicit VI computationally feasible for models with high-dimensional latent variables.

Original: We successfully applied this approach to Bayesian neural networks and achieved superior performance on both regression and classiﬁcation tasks.
Revised: We successfully applied this approach to Bayesian neural networks and achieved superior performance on both regression and classiﬁcation tasks.

Original: We also demonstrate that KIVI can be applied to learn local latent variable models like VAEs.
Revised: We also demonstrate that KIVI can be applied to learn local latent variable models like VAEs.

Original: Future work may include applying this method to larger-scale networks and improving the kernel estimator further.
Revised: Future work may include applying this method to larger-scale networks and improving the kernel estimator further.

Original: A P ROOF OF P ROPOSITION
Revised: A P ROOF OF P ROPOSITION

Original: Proof.
Revised: Proof.

Original: Let V denote the span of the representers of the sample points:
Revised: Let V denote the span of the representers of the sample points:

Original: [Equation]
Revised: V = span( { k ( z i , · ) : i = 1 , . . . , n p + n q } =   n p (cid:88) i

Original: [Equation]
Revised: = α i k

Original:  where we label z p 1: n p as z 1: n p and z q 1: n q as z n p +1: n p + n q .
Revised: ( z pi , · ) + n q (cid:88) j = β j k ( z qj , · )   , where we label z p 1: n p as z 1: n p and z q 1: n q as z n p +1: n p + n q .

Original: Deﬁne the orthogonal complement V ? to be
Revised: Deﬁne the orthogonal complement V ⊥ to be

Original: [Equation]
Revised: V ⊥ =

Revised: { f ∈ H : (cid:104) f, g (cid:105) = 0 , ∀ g ∈ V } .

Original: Because ^ r 2 H , it can be decomposed into two parts:
Revised: Because ˆ r ∈ H , it can be decomposed into two parts:

Original: [Equation]
Revised: [Equation]

Original: where ^ r ∥ 2 V , and ^ r ? 2 V ? .
Revised: where ˆ r (cid:107) ∈ V , and ˆ r ⊥ ∈ V ⊥ .

Original: Note that the squared loss is not changed by the orthogonal component ^ r ? :
Revised: Note that the squared loss is not changed by the orthogonal component ˆ r ⊥ :

Original: [Equation]
Revised: [Equation]

Original: Meanwhile the regularization term can be decomposed into ∥ ^ r ∥ ∥ 2 H + ∥ ^ r ? ∥ 2 H .
Revised: Meanwhile the regularization term can be decomposed into (cid:107) ˆ r (cid:107) (cid:107) 2 H + (cid:107) ˆ r ⊥ (cid:107) 2 H .

Original: So the optimal solution will have ^ r (cid:3)? = 0 , which indicates that ^ r (cid:3) 2 V .
Revised: So the optimal solution will have ˆ r ∗⊥ = 0 , which indicates that ˆ r ∗ ∈ V .

Original: B D ERIVATION OF THE O PTIMAL D ENSITY R ATIO F UNCTION
Revised: B D ERIVATION OF THE O PTIMAL D ENSITY R ATIO F UNCTION

Original: Substituting Eq.
Revised: Substituting Eq.

Original: (5) into Eq.
Revised: (5) into Eq.

Original: (4) and removing the constant, we get
Revised: (4) and removing the constant, we get

Original: [Equation]
Revised: [Equation]

Original: where ( K p ) i;j = 
Revised: where ( K p ) i,j = k

Original: k ( z pi ; z pj ) , ( K pq ) i;j
Revised: ( z pi , z pj ) , ( K pq ) i,j = k

Original: = k ( z pi ; z qj ) , and ( K q ) i;j = 
Revised: ( z pi , z qj ) , and ( K q ) i,j = k

Original: k ( z qi ; z qj ) .
Revised: ( z qi , z qj ) .

Original: Taking derivatives w.r.t.
Revised: Taking derivatives w

Original: (cid:11) and (cid:12) and setting them to be zeros:
Revised: [Equation] r [Equation] t. α and β and setting them to be zeros:

Original: [Equation]
Revised: [Equation]

Original: Rearranging the terms, we have(
Revised: Rearranging the terms, we have

Original: [Equation]
Revised: [Equation]

Original: Left multiplying Eq.
Revised: Left multiplying Eq.

Original: (8) with K ⊤ pq K (cid:0) 1 p and then subtracting Eq.
Revised: (8) with K (cid:62) pq K − 1 p and then subtracting Eq.

Original: (9) yields
Revised: (9) yields

Original: [Equation]
Revised: [Equation]

Original: Substituting the optimal (cid:12) into Eq.
Revised: Substituting the optimal β into Eq.

Original: (8), we get
Revised: (8), we get

Original: [Equation]
Revised: [Equation]

Original: Note that although K (cid:0) 1 p is used in the derivation, the forms of the solution do not require K p to be invertible.
Revised: Note that although K − 1 p is used in the derivation, the forms of the solution do not require K p to be invertible.

Original: In fact, if K p has zero eigenvalues (not invertible), the objective L ( (cid:11) ; (cid:12) ) is not bounded below since one can always choose an (cid:11) in the null space of K p to decrease it.
Revised: In fact, if K p has zero eigenvalues (not invertible), the objective L ( α , β ) is not bounded below since one can always choose an α in the null space of K p to decrease it.

Original: In other words, the form of the solution is well deﬁned in any condition and is optimal when the objective is well deﬁned.
Revised: In other words, the form of the solution is well deﬁned in any condition and is optimal when the objective is well deﬁned.

Original: C G RADIENTS OF THE KL T ERM
Revised: C G RADIENTS OF THE KL T ERM

Original: [Equation]
Revised: [Equation]

Original: The formula above shows that a good density ratio estimator gives accurate gradient estimation of ϕ .
Revised: The formula above shows that a good density ratio estimator gives accurate gradient estimation of φ .

Original: D KIVI WITH A DAPTIVE C ONTRAST
Revised: D KIVI WITH A DAPTIVE C ONTRAST

Original: Although KIVI gives rather good estimation of the density ratio, the estimation accuracy still degrades with larger discrepancy between p and q .
Revised: Although KIVI gives rather good estimation of the density ratio, the estimation accuracy still degrades with larger discrepancy between p and q .

Original: The problem is very critical for local latent variable models like VAEs because the same variational model is required to infer posteriors of all local latent variables.
Revised: The problem is very critical for local latent variable models like VAEs because the same variational model is required to infer posteriors of all local latent variables.

Original: In order to mitigate that, AVB (Mescheder et al., 2017) adopted a technique called Adaptive Contrast (AC), which can easily be integrated with KIVI.
Revised: In order to mitigate that, AVB (Mescheder et al., 2017) adopted a technique called Adaptive Contrast (AC), which can easily be integrated with KIVI.

Original: AC introduces an auxiliary tractable distribution r (cid:11) ( z j x ) that resembles q .
Revised: AC introduces an auxiliary tractable distribution r α ( z | x ) that resembles q .

Original: With r (cid:11) ( z j x ) , the ELBO can be rewritten as
Revised: With r α ( z | x ) , the ELBO can be rewritten as

Original: [Equation]
Revised: [Equation]

Original: Gradients of the ﬁrst term w.r.t.
Revised: Gradients of the ﬁrst term w.r.t.

Original: ϕ can be easily computed using Monte Carlo, and gradients of the second term can be estimated using KIVI.
Revised: φ can be easily computed using Monte Carlo, and gradients of the second term can be estimated using KIVI.

Original: Adaptive contrast gives better estimates if r (cid:11) ( z j x ) approximates q well.
Revised: Adaptive contrast gives better estimates if r α ( z | x ) approximates q well.

Original: Because r is required to have a tractable density, a commonly used adaptive distribution is a Gaussian distribution whose mean (cid:22) r and standard derivation (cid:27) r match with q .
Revised: Because r is required to have a tractable density, a commonly used adaptive distribution is a Gaussian distribution whose mean µ r and standard derivation σ r match with q .

Original: In practice (cid:22) r and (cid:27) r are estimated from the samples of q .
Revised: In practice µ r and σ r are estimated from the samples of q .

Original: According to the invariance of KL divergence under reparameterization, we have
Revised: According to the invariance of KL divergence under reparameterization, we have

Original: [Equation]
Revised: [Equation]

Original: where ^ q ϕ ( z j x ) denotes the distribution of ^
Revised: where ˆ q φ ( z | x ) denotes the distribution of ˆ z =

Original: z = z (cid:0) (cid:22) r (cid:27) r and ^ r 0 (^{+
 (ˆ+} z ) denotes the standard normal distribution.
Revised: z − µ r σ r and ˆ r {+
 (ˆ+} z ) denotes the standard normal distribution.

Original: Under this reparameterization, we only need to estimate the density ratio between two distributions with zero means and unit variances.
Revised: Under this reparameterization, we only need to estimate the density ratio between two distributions with zero means and unit variances.

Original: E L OWER B OUND WITH G AMMA -P RIOR P RECISION
Revised: E L OWER B OUND WITH G AMMA -P RIOR P RECISION

Original: In the multivariate regression task, the output is sampled from a normal distribution with ^ y ( x; W ) as the mean and a parameter as the variance.
Revised: In the multivariate regression task, the output is sampled from a normal distribution with ˆ y ( x, W ) as the mean and a parameter as the variance.

Original: The variance controls the likelihood of the model, therefore, choosing an appropriate variance is essential.
Revised: The variance controls the likelihood of the model, therefore, choosing an appropriate variance is essential.

Original: We place a Gamma prior Gamma (6 ; 6) on its reciprocal (i.e., the precision of the normal distribution).
Revised: We place a Gamma prior Gamma (6 , 6) on its reciprocal (i.e., the precision of the normal distribution).

Original: The variational posterior we used is q [Equation]
Revised: The variational posterior we used is q ( W , λ )

Original: [Equation] ) =q [Equation]
Revised: =

Original:  W ) q [Equation] ) ; q ( [Equation]
Revised: q ( W ) q ( λ ) , q ( λ ) ∼ Gamma( α, β ) .

Original: ) (cid:24) Gamma( (cid:11); (cid:12) ) .

Original: Then the ELBO can be calculated as
Revised: Then the ELBO can be calculated as

Original: [Equation]
Revised: [Equation]

Original: where  ( x ) is the digamma function and KL ( q ( (cid:21) ) ∥ p ( (cid:21) )) can be calculated in closed-form.
Revised: where ψ ( x ) is the digamma function and KL ( q ( λ ) (cid:107) p ( λ )) can be calculated in closed-form.

Original: F A DDITIONAL E XPERIMENT R ESULTS F. 2-D B AYESIAN L OGISTIC R EGRESSION
Revised: F A DDITIONAL E XPERIMENT R ESULTS F. 2-D B AYESIAN L OGISTIC R EGRESSION

Original: [Figure]
Revised: [Figure]

Original: We also conduct experiments on a 2-D Bayesian logistic regression example, which has an intractable posterior.
Revised: We also conduct experiments on a 2-D Bayesian logistic regression example, which has an intractable posterior.

Original: The model is
Revised: The model is

Original: [Equation]
Revised: [Equation]

Original: where w ; x i 2 R 2 ; (cid:27) is the sigmoid function.
Revised: where w , x i ∈ R 2 ; σ is the sigmoid function.

Original: N = 200 data points ( f ( x i ; y i ) 
Revised: N = 200 data points ( { ( x i , y i ) }

Original: g Ni =1 ) are generated from the true model as the training data (Fig. 5a).
Revised: Ni =1 ) are generated from the true model as the training data (Fig. 5a).

Original: The unnormalized true posterior is plotted in
Revised: The unnormalized true posterior is plotted in

Original: Fig.
Revised: Fig.

Original: 5b.
Revised: 5b.

Original: As a baseline, we ﬁrst run VI with a factorized normal distribution.
Revised: As a baseline, we ﬁrst run VI with a factorized normal distribution.

Original: The result is shown in
Revised: The result is shown in

Original: Fig.
Revised: Fig.

Original: 5c.
Revised: 5c.

Original: It can be clearly seen that the factorized normal can capture the position and the scale of the true posterior but cannot ﬁt well to the shape due to its independence across dimensions.
Revised: It can be clearly seen that the factorized normal can capture the position and the scale of the true posterior but cannot ﬁt well to the shape due to its independence across dimensions.

Original: We then apply KIVI.
Revised: We then apply KIVI.

Original: The implicit posterior we use is a simple stochastic neural network (see Appendix G.1).
Revised: The implicit posterior we use is a simple stochastic neural network (see Appendix G.1).

Original: To demonstrate how good the result is, we also run Hamiltonian Monte Carlo (HMC) to get posterior samples.
Revised: To demonstrate how good the result is, we also run Hamiltonian Monte Carlo (HMC) to get posterior samples.

Original: The results are plotted in Fig.
Revised: The results are plotted in Fig.

Original: 5d and 5e.
Revised: 5d and 5e.

Original: We can see that the implicit posterior is learned to capture the strong correlation between the two dimensions and can produce posterior samples that have a similar shape with the samples drawn by HMC.
Revised: We can see that the implicit posterior is learned to capture the strong correlation between the two dimensions and can produce posterior samples that have a similar shape with the samples drawn by HMC.

Original: [Figure]
Revised: [Figure]

Original: We also use this experiment to illustrate the importance of the reverse ratio trick.
Revised: We also use this experiment to illustrate the importance of the reverse ratio trick.

Original: See Figure 6 for the implicit variational distribution learned by only optimizing KL( q ϕ ( w ) ∥ p ( w )) .
Revised: See Figure 6 for the implicit variational distribution learned by only optimizing KL( q φ ( w ) (cid:107) p ( w )) .

Original: In this way we expect the learned q to be close to the prior N ( 0 ; I ) .
Revised: In this way we expect the learned q to be close to the prior N ( 0 , I ) .

Original: The result produced by using the trick is compared to the result without it.
Revised: The result produced by using the trick is compared to the result without it.

Original: We can see that the latter fails to work well.
Revised: We can see that the latter fails to work well.

Original: F. C OMPARISON WITH M ORE M ETHODS
Revised: F. C OMPARISON WITH M ORE M ETHODS

Original: In this section we present comparisons with two more methods towards ﬂexible posteriors, namely, normalizing ﬂow (Rezende & Mohamed, 2015) and KSD variational inference (KSD VI) (Liu &
Revised: In this section we present comparisons with two more methods towards ﬂexible posteriors, namely, normalizing ﬂow (Rezende & Mohamed, 2015) and KSD variational inference (KSD VI) (Liu &

Original: Feng, 2016).
Revised: Feng, 2016).

Original: We also include the results by HMC as the ground truth.
Revised: We also include the results by HMC as the ground truth.

Original: Normalizing ﬂow The basic idea of normalizing ﬂow has been introduced in Section 1.
Revised: Normalizing ﬂow The basic idea of normalizing ﬂow has been introduced in Section 1.

Original: Speciﬁcally, given a random variable z 2 R d following a simple distribution q ( z ) , and an invertible mapping
Revised: Speciﬁcally, given a random variable z ∈ R d following a simple distribution q ( z ) , and an invertible mapping

Original: T : R d ! R d so that z ′ =
Revised: T : R d → R d so that z (cid:48) =

Original: T ( z ) , the probability density of the transformed variable z ′ can be calculated as
Revised: T ( z ) , the probability density of the transformed variable z (cid:48) can be calculated as

Original: [Equation]
Revised: [Equation]

Original: Thus we can construct complex distributions by composing invertible mappings ( T ), while keeping the probability density of the result distribution tractable by successively applying Eq.
Revised: Thus we can construct complex distributions by composing invertible mappings ( T ), while keeping the probability density of the result distribution tractable by successively applying Eq.

Original: For
Revised: For

Original: [Table]
Revised: [Table]

Original: example, Rezende & Mohamed (2015) proposed the planar normalizing ﬂow:
Revised: example, Rezende & Mohamed (2015) proposed the planar normalizing ﬂow:

Original: [Equation]
Revised: [Equation]

Original: where u ; w 2 R d ; b 2 R are free parameters, and h is a smooth element-wise nonlinearty, chosen as Tanh in the following experiments.
Revised: where u , w ∈ R d , b ∈ R are free parameters, and h is a smooth element-wise nonlinearty, chosen as Tanh in the following experiments.

Original: A simple variational posterior can be made more expressive when this parametric transformation is employed.
Revised: A simple variational posterior can be made more expressive when this parametric transformation is employed.

Original: KSD VI KSD variational inference ( Liu & Feng, 2016) is a method that directly minimizes the kernelized Stein discrepancy (KSD) between the true posterior ( p ) and the variational posterior ( q ):
Revised: KSD VI KSD variational inference (Liu & Feng, 2016) is a method that directly minimizes the kernelized Stein discrepancy (KSD) between the true posterior ( p ) and the variational posterior ( q ):

Original: [Equation]
Revised: [Equation]

Original: (11) where H d is a unit ball in the vector-valued RKHS induced by the RBF kernel k ( z ; z′ )
Revised: 11) where H d is a unit ball in the vector-valued RKHS induced by the RBF kernel k ( z , z

Original: = ∥
Revised: [Equation]

Original:  z (cid:0) z ′ ∥ 22 2 (cid:27) 2 , and s q ( z ) = ∇ z log p ( z j x ) .
Revised: ) = (cid:107) z − z (cid:48) (cid:107) 22 2 σ 2 , and s q ( z ) = ∇ z log p ( z | x ) .

Original: Note that the objective in Eq.
Revised: Note that the objective in Eq.

Original: ( 11) only requires samples from q , so 
Revised: (11) only requires samples from q , so KSD

Original: KSD VI also belongs to implicit VI methods.
Revised: VI also belongs to implicit VI methods.

Original: We conduct the regression experiments in Section 6.2.1.
Revised: We conduct the regression experiments in Section 6.2.1.

Original: The results are shown in Table 2.
Revised: The results are shown in Table 2.

Original: For normalizing ﬂow, we apply 10 planar ﬂows on the weights to match the running time of our implicit posteriors.
Revised: For normalizing ﬂow, we apply 10 planar ﬂows on the weights to match the running time of our implicit posteriors.

Original: To see whether ﬂows help the inference, we also present results of VI using factorized normal distributions on the weights (with their means and standard deviations optimized by the reparameterization trick).
Revised: To see whether ﬂows help the inference, we also present results of VI using factorized normal distributions on the weights (with their means and standard deviations optimized by the reparameterization trick).

Original: For KSD VI, we use the same implicit posteriors as those we used for
Revised: For KSD VI, we use the same implicit posteriors as those we used for

Original: KIVI in Section 6.2.1.
Revised: KIVI in Section 6.2.1.

Original: For VI with factorized normal distributions and normalizing ﬂow, we usesamples, batch size 10 except that for kin8nm and naval we use batch size 100.
Revised: For VI with factorized normal distributions and normalizing ﬂow, we use 100 samples, batch size 10 except that for kin8nm and naval we use batch size 100.

Original: We set the learning rate as 0.01 and run 500 epochs for 20 times.
Revised: We set the learning rate as 0.01 and run 500 epochs for 20 times.

Original: For KSD VI, we set all the training parameters (number of samples, number of epochs, batch size, and learning rate) the same as KIVI’s.
Revised: For KSD VI, we set all the training parameters (number of samples, number of epochs, batch size, and learning rate) the same as KIVI’s.

Original: We also run HMC to produce the ground truths.
Revised: We also run HMC to produce the ground truths.

Original: For HMC, we use 20 chains, 150000 iterations and set the target acceptance rate as 97% to adapt the step size.
Revised: For HMC, we use 20 chains, 150000 iterations and set the target acceptance rate as 97% to adapt the step size.

Original: As the HMC experiment is time-consuming, we only perform 10 runs.
Revised: As the HMC experiment is time-consuming, we only perform 10 runs.

Original: From Table 2 we can see that normalizing ﬂow does not show improvements over VI with factorized normal distributions.
Revised: From Table 2 we can see that normalizing ﬂow does not show improvements over VI with factorized normal distributions.

Original: This is probably due to optimization challenges caused by the limited form of planar ﬂows, thus the inference procedure doesn’t take much beneﬁt from the ﬂexibility introduced by the ﬂow.
Revised: This is probably due to optimization challenges caused by the limited form of planar ﬂows, thus the inference procedure doesn’t take much beneﬁt from the ﬂexibility introduced by the ﬂow.

Original: For KSD VI, we found it is very sensitive to the initialization of implicit posteriors.
Revised: For KSD VI, we found it is very sensitive to the initialization of implicit posteriors.

Original: We had to set the variance of the input Gaussian noise larger so that KSD VI wouldn’t diverge at the very beginning of training.
Revised: We had to set the variance of the input Gaussian noise larger so that KSD VI wouldn’t diverge at the very beginning of training.

Original: KSD VI also soon converges to unsatisfying local optima after the optimization process starts.
Revised: KSD VI also soon converges to unsatisfying local optimas after the optimization process starts.

Original: These two ﬁndings are well explained by the conclusion that KSD is the magnitude of a functional gradient of KL divergence (Liu & Wang, 2016), then all saddle points in the original problem of optimizing the KL divergence become local optima when optimizingthe KSD.
Revised: These two ﬁndings are well explained by the conclusion that KSD is the magnitude of a functional gradient of KL divergence (Liu & Wang, 2016), then all saddle points in the original problem of optimizing the KL divergence become local optimas when optimizing KSD.

Original: F. V ISUALIZATION OF I NFERRED P OSTERIORS
Revised: F. V ISUALIZATION OF I NFERRED P OSTERIORS

Original: In order to visually check the quality of the uncertainty estimated by KIVI, we use a visualization technique named parallel coordinates (Inselberg & Dimsdale, 1987) to plot the posterior over weights.
Revised: In order to visually check the quality of the uncertainty estimated by KIVI, we use a visualization technique named parallel coordinates (Inselberg & Dimsdale, 1987) to plot the posterior over weights.

Original: We compare the posteriors inferred in the regression experiment on Boston housing (Table 2) by VI with factorized normal approximation, HMC, and KIVI.
Revised: We compare the posteriors inferred in the regression experiment on Boston housing (Table 2) by VI with factorized normal approximation, HMC, and KIVI.

Original: The feature dimension of the Boston housing data is 13.
Revised: The feature dimension of the Boston housing data is 13.

Original: And the BNN used is an MLP with a hidden layer of size 50.
Revised: And the BNN used is an MLP with a hidden layer of size 50.

Original: So the network has two weight matrices: W 0 2 R 14 (cid:2) 50 and W 1 2 R 51 (cid:2) 1 (The bias parameters are included).
Revised: So the network has two weight matrices: W 0 ∈ R 14 × 50 and W 1 ∈ R 51 × 1 (The bias parameters are included).

Original: Since W 0 has 700 dimensions, we only plot the posterior samples of Wto avoid visual clutter.
Revised: Since W 0 has 700 dimensions, we only plot the posterior samples of Wto avoid visual clutter.

Original: Speciﬁcally, we draw 100 samples from the posterior: W (1:100) (cid:24) q ( W 1 ) , and the goal is to show these 51-dimensional samples.
Revised: Speciﬁcally, we draw 100 samples from the posterior: W (1:100) ∼ q ( W 1 ) , and the goal is to show these 51-dimensional samples.

Original: In parallel coordinates each sample is plotted as a polyline (see Figure 7).
Revised: In parallel coordinates each sample is plotted as a polyline (see Figure 7).

Original: The vertices on the polyline represent each single dimension.
Revised: The vertices on the polyline represent each single dimension.

Original: Their positions on the horizontal axis are the indices of the dimension (from 0 to 50).
Revised: Their positions on the horizontal axis are the indices of the dimension (from 0 to 50).

Original: And the vertical coordinates represent the weight values.
Revised: And the vertical coordinates represent the weight values.

Original: Note that an important fact about hidden neurons in neural networks is that they are non-identiﬁable (any two hidden nodes can be exchanged without affecting the output distribution).
Revised: Note that an important fact about hidden neurons in neural networks is that they are non-identiﬁable (any two hidden nodes can be exchanged without affecting the output distribution).

Original: This indicates that all dimensions of W 1 are non-identiﬁable.
Revised: This indicates that all dimensions of W 1 are non-identiﬁable.

Original: Different inference algorithms may converge to different local modes caused by this symmetry.
Revised: Different inference algorithms may converge to different local modes caused by this symmetry.

Original: To make the visualizations comparable, we sort the dimensions of W 1 for each algorithm by the mean value of posterior samples in each dimension.
Revised: To make the visualizations comparable, we sort the dimensions of W 1 for each algorithm by the mean value of posterior samples in each dimension.

Original: From the visualization we could see that BNNs with factorized normal approximation have signiﬁcant over-pruning problems.
Revised: From the visualization we could see that BNNs with factorized normal approximation have signiﬁcant over-pruning problems.

Original: A large proportion of hidden nodes are turned off during the inference and the information is mainly carried by several others.
Revised: A large proportion of hidden nodes are turned off during the inference and the information is mainly carried by several others.

Original: These pruned weights can be identiﬁed by their low signal-to-noise ratio j (cid:22) j (cid:27) , where (cid:22) is the mean, and (cid:27) is the standard deviation.
Revised: These pruned weights can be identiﬁed by their low signal-to-noise ratio | µ | σ , where µ is the mean, and σ is the standard deviation.

Original: This problem has been pointed out previously, both in VAE works ( Sønderby et al., 2016; Hoffman, 2017) and BNN works (Blundell et al., 2015), and can be explained by the looseness of the variational bound when factorized approximation is used (Trippe & Turner, 2017).
Revised: This problem has been pointed out previously, both in VAE works (Sønderby et al., 2016; Hoffman, 2017) and BNN works (Blundell et al., 2015), and can be explained by the looseness of the variational bound when factorized approximation is used (Trippe & Turner, 2017).

Original: In contrast, the ground truth by HMC doesn’t have the pruning problems.
Revised: In contrast, the ground truth by HMC doesn’t have the pruning problems.

Original: And there are very strong correlations captured if we observe the neighboring dimensions.
Revised: And there are very strong correlations captured if we observe the neighboring dimensions.

Original: 2 KIVI also doesn’t have the pruning problems and could capture the strong correlations across the dimensions.
Revised: 2 KIVI also doesn’t have the pruning problems and could capture the strong correlations across the dimensions.

Original: And with the gain in accuracy, there is still a good amount of uncertainty in the implicit posterior.
Revised: And with the gain in accuracy, there is still a good amount of uncertainty in the implicit posterior.

Original: Despite the weight dimensions are non-identiﬁable, we could still see that the two VI methods arrive at biased solutions compared to the ground truth by HMC in terms of scales.
Revised: Despite the weight dimensions are non-identiﬁable, we could still see that the two VI methods arrive at biased solutions compared to the ground truth by HMC in terms of scales.

Original: Note that this is not necessarily problematic since VI is typically known to produce biased solutions.
Revised: Note that this is not necessarily problematic since VI is typically known to produce biased solutions.

Original: F. A CCURACY OF KL E STIMATION
Revised: F. A CCURACY OF KL E STIMATION

Original: From Section 3.1 we know that the approximate gradients of the ELBO are directly related to the KL estimates.
Revised: From Section 3.1 we know that the approximate gradients of the ELBO are directly related to the KL estimates.

Original: So we would like to assess the accuracy of the KL estimator.
Revised: So we would like to assess the accuracy of the KL estimator.

Original: We adopt the settings from the VAE experiment on MNIST (Section 6.3).
Revised: We adopt the settings from the VAE experiment on MNIST (Section 6.3).

Original: For comparison, we must also be able to compute a ground truth of the KL term.
Revised: For comparison, we must also be able to compute a ground truth of the KL term.

Original: To achieve this, we use normalizing ﬂow in q ( z j x ) , which has a complicated but tractable density.
Revised: To achieve this, we use normalizing ﬂow in q ( z | x ) , which has a complicated but tractable density.

Original: Thus we can get a good Monte Carlo estimate of the true KL term: KL( q ϕ ( z j x ) ∥ p ( z )) ≃ 1 m ∑ mi =1 log q ϕ ( z i j x ) p ( z i ) ;
Revised: Thus we can get a good Monte Carlo estimate of the true KL term: KL( q φ ( z | x ) (cid:107) p ( z )) (cid:39) 1 m (cid:80) mi =1 log q φ ( z i | x ) p ( z i ) ,

Original: z i (cid:24) q ϕ ( z j x ) :
Revised: z i ∼ q φ ( z | x ) .

Original: In Figure 8a we compare the KL term estimated using KIVI with the ground truth.
Revised: In Figure 8a we compare the KL term estimated using KIVI with the ground truth.

Original: Note that since we use adaptive-contrast (AC) (see
Revised: Note that since we use adaptive-contrast (AC) (see

Original: Appendix D) in the VAE experiments, where the KL term is broken down into two parts, it should make more sense to look at the only part that uses the density ratio estimator (the KL divergence between the standardized q and a standard normal distribution), which is plotted in Figure 8b.
Revised: Appendix D) in the VAE experiments, where the KL term is broken down into two parts, it should make more sense to look at the only part that uses the density ratio estimator (the KL divergence between the standardized q and a standard normal distribution), which is plotted in Figure 8b.

Original: We can see that the KL estimates closely track the ground truth, and are more accurate as the variational approximation improves over time.
Revised: We can see that the KL estimates closely track the ground truth and are more accurate as the variational approximation improves over time

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: F. C HANGE OF I NTERPOLATION R ESULTS ON C ELEB A T HROUGH T RAINING
Revised: F. C HANGE OF I NTERPOLATION R ESULTS ON C ELEB A T HROUGH T RAINING

Original: In Figures 9 to 14 we present the generated images for the interpolation experiments on CelebA through the training process.
Revised: In Figures 9 to 14 we present the generated images for the interpolation experiments on CelebA through the training process.

Original: The images are generated after 1, 5, 10, 15, 20, and 25 epochs.
Revised: The images are generated after 1, 5, 10, 15, 20, and 25 epochs.

Original: Results on the left are produced by AVB, and on the right by KIVI.
Revised: Results on the left are produced by AVB, and on the right by KIVI.

Original: It can be clearly seen that AVB’s training process is of very high variance, as we have mentioned in Section 2.
Revised: It can be clearly seen that AVB’s training process is of very high variance, as we have mentioned in Section 2.

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: G D ETAILS OF E XPERIMENTS G. T OY E XPERIMENTS
Revised: G D ETAILS OF E XPERIMENTS G. T OY E XPERIMENTS

Original: 1-D Gaussian Mixture
Revised: 1-D Gaussian Mixture

Original: The implicit posterior generates samples by propagating samples from a standard normal distribution through a two-layer MLP with 10 hidden units and one output unit.
Revised: The implicit posterior generates samples by propagating samples from a standard normal distribution through a two-layer MLP with 10 hidden units and one output unit.

Original: We set the regularization coefﬁcient (cid:21) as 0 : 003 and the density ratio clipping threshold as 10 (cid:0) 8 .
Revised: We set the regularization coefﬁcient λ as 0 . 003 and the density ratio clipping threshold as 10 − 8 .

Original: 2-D Bayesian Logistic Regression The inputs X are 200 points randomly sampled from U[ (cid:0) 5 ; 5] (cid:2)
Revised: 2-D Bayesian Logistic Regression The inputs X are 200 points randomly sampled from U[ − 5 , 5] ×

Original: U[ (cid:0) 5 ; 5] .
Revised: U[ − 5 , 5] .

Original: The outputs Y are the predictions of X with randomly sampled weights from the prior.
Revised: The outputs Y are the predictions of X with randomly sampled weights from the prior.

Original: For HMC, we run 100 chains, 200 iterations each, and use 10 leapfrog steps.
Revised: For HMC, we run 100 chains, 200 iterations each, and use 10 leapfrog steps.

Original: The step size is automatically adapted using dual averaging, starting from 0.001.
Revised: The step size is automatically adapted using dual averaging, starting from 0.001.

Original: We discard the ﬁrst 100 samples generated.
Revised: We discard the ﬁrst 100 samples generated.

Original: For factorized VI, the training is run for 100 epochs and 100 samples are used.
Revised: For factorized VI, the training is run for 100 epochs and 100 samples are used.

Original: In the training, we anneal the learning rate linearly according to lr =
Revised: In the training, we anneal the learning rate linearly according to lr =

Original: [Figure]
Revised: [Figure]

Original: For KIVI we follow the same setting, except that we use 1k samples.
Revised: For KIVI we follow the same setting, except that we use 1k samples.

Original: The regularization coefﬁcient (cid:21) and the minimal density ratio are set to 0.1 and 10 (cid:0) 8 , respectively.
Revised: The regularization coefﬁcient λ and the minimal density ratio are set to 0.1 and 10 − 8 , respectively.

Original: Below we describe the sample generation process of the
Revised: Below we describe the sample generation process of the

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: implicit variational posterior used in KIVI.
Revised: implicit variational posterior used in KIVI.

Original: First, some 2-D random normal samples are propagated through two fully-connected layers of size 20, producing h (we don’t use activation functions for the second layer), and then h is added with another random normal noise with trainable variances, producing z .
Revised: First, some 2-D random normal samples are propagated through two fully-connected layers of size 20, producing h (we don’t use activation functions for the second layer), and then h is added with another random normal noise with trainable variances, producing z .

Original: Finally, we propagate z through a fully-connected layer of size 20 and then a linear output layer, getting the variational samples.
Revised: Finally, we propagate z through a fully-connected layer of size 20 and then a linear output layer, getting the variational samples.

Original: [Figure]
Revised: [Figure]

Original: G. B AYESIAN N EURAL N ETWORKS G.2. R EGRESSION
Revised: G. B AYESIAN N EURAL N ETWORKS G.2. R EGRESSION

Original: As the regression datasets have small feature dimensions (all less than 15, except 90 for Year ), using
Revised: As the regression datasets have small feature dimensions (all less than 15, except 90 for Year ), using

Original: BNNs of one hidden layer (50 units) doesn’t produce very high-dimensional weights.
Revised: BNNs of one hidden layer (50 units) doesn’t produce very high-dimensional weights.

Original: Therefore, we still use MLPs in the implicit variational posterior, of which the samples are generated by propagating samples from a standard normal distribution through an MLP.
Revised: Therefore, we still use MLPs in the implicit variational posterior, of which the samples are generated by propagating samples from a standard normal distribution through an MLP.

Original: For all datasets, we use ReLU as the activation function.
Revised: For all datasets, we use ReLU as the activation function.

Original: The MLP has one hidden layer except that for Yacht it has two hidden layers.
Revised: The MLP has one hidden layer except that for Yacht it has two hidden layers.

Original: We list the details in Table 3, which consist of 10 datasets and 2 weight matrices each.
Revised: We list the details in Table 3, which consist of 10 datasets and 2 weight matrices each.

Original: Taking Layer 1 for Boston as an example, (20, 30, N1) represents that 20 random normal samples are generated and propagated through an MLP with a hidden layer of size 30 and an output layer of size N1 = 14 (cid:2) 50 .
Revised: Taking Layer 1 for Boston as an example, (20, 30, N1) represents that 20 random normal samples are generated and propagated through an MLP with a hidden layer of size 30 and an output layer of size N1 = 14 × 50 .

Original: Note that N1 corresponds to the number of weights in the ﬁrst layer of the BNN.
Revised: Note that N1 corresponds to the number of weights in the ﬁrst layer of the BNN.

Original: [Table]
Revised: [Table]

Original: [Table] Layer(100, 500, N1) (100, 500, N1) (20, 10, N1) (
Revised: Layer

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Layer(100, 500, N2) (

Original: [Table]
Revised: [Table]

Original: 500, N2) (
Revised: N1) (

Original: [Table] [Table]
Revised: [Table] [Table]

Original: N2)
Revised: N1)

Original: [Table] [Table] 
Revised: [Table] [Table] [Table]

Original: N2)
Revised: N

Original: [Table]
Revised: [Table]

Revised: Layer

Original: G.2.
Revised: [Table] [Table] [Table] N

Original: [Table]
Revised: [Table]

Original: C LASSIFICATION
Revised: C LASSIFICATION

Original: MNIST classiﬁcation needs a larger scale network than the one used in multivariate regression.
Revised: MNIST classiﬁcation needs a larger scale network than the one used in multivariate regression.

Original: Therefore, we adopt an MMNN in the implicit variational posterior.
Revised: Therefore, we adopt an MMNN in the implicit variational posterior.

Original: We denote the hidden-layer size of the BNN as L (In our experiments L = 400 ; 800 or 1200 ).
Revised: We denote the hidden-layer size of the BNN as L (In our experiments L = 400 , 800 or 1200 ).

Original: Below we set N =500 when
Revised: Below we set N =

Original:  L = 400 , otherwise we set N = 800 .
Revised: 500 when L = 400 , otherwise we set N = 800 .

Original: In the MMNN, we use two matrix multiplication layers.
Revised: In the MMNN, we use two matrix multiplication layers

Original: For the ﬁrst-layer weights ( 785 (cid:2) L ), the two hidden matrix multiplication layers are both of size N (cid:2) N and are with ReLU activations.
Revised: For the ﬁrst-layer weights ( 785 × L ), the two hidden matrix multiplication layers are both of size N × N and are with ReLU activations.

Original: The output layer of the MMNN is of size L (cid:2) 785 and is with linear activations.
Revised: The output layer of the MMNN is of size L × 785 and is with linear activations.

Original: The input matrices are random samples of size 30 (cid:2) 30 drawn from a standard normal distribution.
Revised: The input matrices are random samples of size 30 × 30 drawn from a standard normal distribution.

Original: For the second-layer weights ( L (cid:2) ( L + 1) ), the two hidden matrix multiplication layers are both of size N (cid:2) N and are with ReLU activations.
Revised: For the second-layer weights ( L × ( L + 1) ), the two hidden matrix multiplication layers are both of size N × N and are with ReLU activations.

Original: The output layer of the MMNN is of size L (cid:2) ( L + 1) and is with linear activations.
Revised: The output layer of the MMNN is of size L × ( L + 1) and is with linear activations.

Original: The input matrices are random samples of size 30 (cid:2) 30 drawn from a standard normal distribution.
Revised: The input matrices are random samples of size 30 × 30 drawn from a standard normal distribution.

Original: For the third-layer weights ( 10 (cid:2) ( L + 1) ), the two hidden matrix multiplication layers are both of size 30 (cid:2) N and are with ReLU activations.
Revised: For the third-layer weights ( 10 × ( L + 1) ), the two hidden matrix multiplication layers are both of size 30 × N and are with ReLU activations.

Original: The output layer of the MMNN is of size 10 (cid:2) ( L + 1) and is with linear activations.
Revised: The output layer of the MMNN is of size 10 × ( L + 1) and is with linear activations.

Original: The input matrices are random samples of size 30 (cid:2) 30 drawn from a standard normal distribution.
Revised: The input matrices are random samples of size 30 × 30 drawn from a standard normal distribution.

Original: We use the above variational posterior settings in both KIVI and PC.
Revised: We use the above variational posterior settings in both KIVI and PC.

Original: For PC, a logistic regression serves as the discriminator.
Revised: For PC, a logistic regression serves as the discriminator.

Original: We set the regularization coefﬁcient (cid:21) as 0.001 and the minimal density ratio as 10 (cid:0) 8 for KIVI.
Revised: We set the regularization coefﬁcient λ as 0.001 and the minimal density ratio as 10 − 8 for KIVI.

Original: Both two methods use 10 samples, and we set the batch size as 100 and the learning rate as 0.001 in training.
Revised: Both two methods use 10 samples, and we set the batch size as 100 and the learning rate as 0.001 in training.

Original: G. V ARIATIONAL A UTOENCODERS
Revised: G. V ARIATIONAL A UTOENCODERS

Original: The decoders used in the MNIST experiment are MLPs with two hidden ReLU layers.
Revised: The decoders used in the MNIST experiment are MLPs with two hidden ReLU layers.

Original: The latent dimension is 8.
Revised: The latent dimension is 8.

Original: Each hidden layer is of size 500.
Revised: Each hidden layer is of size 500.

Original: The implicit posterior is also an MLP with two hidden ReLU layers, with Gaussian noises of 500 dimensions added to the ﬁrst hidden layer.
Revised: The implicit posterior is also an MLP with two hidden ReLU layers, with Gaussian noises of 500 dimensions added to the ﬁrst hidden layer.

Original: The noise has zero means and 500-dimensional trainable variances.
Revised: The noise has zero means and 500-dimensional trainable variances.

Original: Training is with the batch size 128.
Revised: Training is with the batch size 128.

Original: The learning rate is 0.001, which is annealed by a factor of 0.5 every 200 epochs.
Revised: The learning rate is 0.001, which is annealed by a factor of 0.5 every 200 epochs.

Original: The parameters for KIVI in this case are n p = n q = 100 ; M = 1 ; (cid:21) = 0 : 001 , and the clipping value is set to 10 (cid:0) 8 .
Revised: The parameters for KIVI in this case are n p = n q = 100 , M = 1 , λ = 0 . 001 , and the clipping value is set to 10 − 8 .

Original: The decoders used in the CelebA experiment have exactly the same structure with the one used for 64 (cid:2) 64 images in the DCGAN paper (Radford et al., 2015).
Revised: The decoders used in the CelebA experiment have exactly the same structure with the one used for 64 × 64 images in the DCGAN paper (Radford et al., 2015).

Original: The latent dimension is 32.
Revised: The latent dimension is 32.

Original: The implicit variational posterior is a deep convolutional neural network with a symmetric structure to the decoder, except that the output of the last convolutional layer is ﬂattened and is added a Gaussian noise of the same shape as the last dimension.
Revised: The implicit variational posterior is a deep convolutional neural network with a symmetric structure to the decoder, except that the output of the last convolutional layer is ﬂattened and is added a Gaussian noise of the same shape as the last dimension.

Original: The noise has zero means and trainable variances.
Revised: The noise has zero means and trainable variances.

Original: The 
Revised: The last hidden layer is fully-connected and has 500 ReLU units. For AVB we use the same decoder.

Original: For  use 
Revised: For both KIVI and AVB, we use batch