Original: Gaussian Density Parametrization Flow: Particle and Stochastic Approaches
Revised: Gaussian Density Parametrization Flow: Particle and Stochastic Approaches

Original: Anonymous Authors

Original: Anonymous Institution

Original: Abstract
Revised: Abstract

Original: Bayesian inference is intractable for most practical problems and requires approximation schemes with several trade-oﬀs.
Revised: Bayesian inference is intractable for most practical problems and requires approximation schemes with several trade-oﬀs.

Original: Variational inference provides one of such approximations which, while powerful, has thus far seen limited use in high-dimensional applications due to its complexity and computational cost.
Revised: Variational inference provides one of such approximations which, while powerful, has thus far seen limited use in high-dimensional applications due to its complexity and computational cost.

Original: This paper introduces a scalable, theoreticallygrounded, and simple-to-implement algorithm for approximate inference with a variational
Revised: This paper introduces a scalable, theoreticallygrounded, and simple-to-implement algorithm for approximate inference with a variational

Original: Gaussian distribution.
Revised: Gaussian distribution.

Original: Speciﬁcally, we establish a practical particle-based algorithm to perform variational Gaussian inference that scales linearly in the problem dimensionality.
Revised: Speciﬁcally, we establish a practical particle-based algorithm to perform variational Gaussian inference that scales linearly in the problem dimensionality.

Original: We show that our approach performs on par with the state of the art on a set of challenging high-dimensional problems.
Revised: We show that our approach performs on par with the state of the art on a set of challenging high-dimensional problems.

Original: 1. Introduction
Revised: 1. Introduction

Original: Computing posterior distributions is intractable for many problems of interest.
Revised: Computing posterior distributions is intractable for many problems of interest.

Original: Markov chain Monte Carlo methods struggle to scale up to large datasets (Bardenet et al., 2017) and convergence is hard to diagnose (Cowles and Carlin, 1996).
Revised: Markov chain Monte Carlo methods struggle to scale up to large datasets (Bardenet et al., 2017) and convergence is hard to diagnose (Cowles and Carlin, 1996).

Original: On the other hand, Variational
Revised: On the other hand, Variational

Original: Inference (VI) methods can rely on well-understood optimization techniques and scale well to large datasets, but the quality of the approximation depends heavily on the assumptions made.
Revised: Inference (VI) methods can rely on well-understood optimization techniques and scale well to large datasets, but the quality of the approximation depends heavily on the assumptions made.

Original: As most interesting real-world problems are typically high-dimensional, VI techniques either make crude approximations regarding the posterior distribution, such as mean-ﬁeld (Hinton and van Camp, 1993)
Revised: As most interesting real-world problems are typically high-dimensional, VI techniques either make crude approximations regarding the posterior distribution, such as mean-ﬁeld (Hinton and van Camp, 1993)

Original: (every variable is assumed to be independent), or do not scale well with the number of dimensions.
Revised: (every variable is assumed to be independent) or do not scale well with the number of dimensions.

Original: An increasingly popular class of solvers are particle-based variational methods (Gershman et al., 2012; Liu and Wang, 2016;
Revised: An increasingly popular class of solvers are particle-based variational methods (Gershman et al., 2012; Liu and Wang, 2016;

Original: Liu et al., 2016; Saeedi et al., 2017).
Revised: Liu et al., 2016; Saeedi et al., 2017).

Original: Lying at the intersection of sampling and variational inference, these methods represent the variational distribution as an empirical distribution.
Revised: Lying at the intersection of sampling and variational inference, these methods represent the variational distribution as an empirical distribution.

Original: They then optimize the particles positions instead of the distribution parameters.
Revised: They then optimize the particles positions instead of the distribution parameters.

Original: In this work we introduce Gaussian Particle Flow (GPF) , a general-purpose algorithm to approximate a Gaussian variational distribution with particles.
Revised: In this work we introduce Gaussian Particle Flow (GPF) , a general-purpose algorithm to approximate a Gaussian variational distribution with particles.

Original: GPF leverages the Gaussian Variational Approximation (GVA) (Opper and Archambeau, 2009), a powerful tool to approximate intractable posterior distributions with multivariate Gaussians.
Revised: GPF leverages the Gaussian Variational Approximation (GVA) (Opper and Archambeau, 2009), a powerful tool to approximate intractable posterior distributions with multivariate Gaussians.

Original: Gaussian Particle Flow (GPF) is free of matrix inversions and second derivatives, scaling linearly with the dimensionality of the problem.
Revised: Gaussian Particle Flow (GPF) is free of matrix inversions and second derivatives, scaling linearly with the dimensionality of the problem.

Original: We show that, under some conditions, the ﬁxed-point solutions for Gaussians are exact and we have linear convergence.
Revised: We show that, under some conditions, the ﬁxed-point solutions for Gaussians are exact and we have linear convergence.

Original: Finally, we provide empirical results both in fully-controlled synthetic settings and on a set of challenging real-world problems, demonstrating that our approach easily scales to high dimensions and achieves state-of-the-art performance.
Revised: Finally, we provide empirical results both in fully-controlled synthetic settings and on a set of challenging real-world problems, demonstrating that our approach easily scales to high dimensions and achieves state-of-the-art performance.

Original: 2. Gaussian Particle Flow 2.1. Particle-Based Variational Inference
Revised: 2. Gaussian Particle Flow 2.1. Particle-Based Variational Inference

Original: Bayesian inference aims to ﬁnd the posterior distribution of a latent variable x given some observations y :
Revised: Bayesian inference aims to ﬁnd the posterior distribution of a latent variable x given some observations y :

Original: [Equation]
Revised: [Equation]

Original: Even if the likelihood p ( y | x ) and the prior p ( x ) are known analytically, marginalizing out high-dimensional variables to compute p ( y ) is typically intractable.
Revised: Even if the likelihood p ( y | x ) and the prior p ( x ) are known analytically, marginalizing out high-dimensional variables to compute p ( y ) is typically intractable.

Original: VI approximates the posterior by the closest distribution within a tractable family, with closeness being measured by the Kullback-Leibler (KL) divergence.
Revised: VI approximates the posterior by the closest distribution within a tractable family, with closeness being measured by the Kullback-Leibler (KL) divergence.

Original: Denoting by Q a family of distributions, we look for arg min q ∈Q KL [ q ( x ) || p
Revised: Denoting by Q a family of distributions, we look for arg min q ∈Q KL [ q ( x ) || p

Original: ( x | y )].
Revised: ( x | y )].

Original: Since p ( y ) is not known, we equivalently minimize the upper bound F :
Revised: Since p ( y ) is not known, we equivalently minimize the upper bound F :

Original: [Equation]
Revised: [Equation]

Original: where H q is the entropy of q .
Revised: where H q is the entropy of q .

Original: Note that F is known as the variational free energy and −F is known as the ELBO.
Revised: Note that F is known as the variational free energy and −F is known as the ELBO.

Original: Stochastic gradient descent methods compute expectations (and gradients) with Monte
Revised: Stochastic gradient descent methods compute expectations (and gradients) with Monte

Original: Carlo samples from the current approximation of the variational density independently at each time step.
Revised: Carlo samples from the current approximation of the variational density independently at each time step.

Original: Particle based methods for variational inference (ParVI) draw samples once at the beginning of the algorithm instead.
Revised: Particle based methods for variational inference (ParVI) draw samples once at the beginning of the algorithm instead.

Original: They iteratively construct a transformation on an initial random variable (with a simple tractable density) leading to the minimization of the variational free energy.
Revised: They iteratively construct a transformation on an initial random variable (with a simple tractable density) leading to the minimization of the variational free energy.

Original: The iterative approach induces a deterministic temporal ﬂow of random variables which depends on the current density of the variable itself.
Revised: The iterative approach induces a deterministic temporal ﬂow of random variables which depends on the current density of the variable itself.

Original: Using an approximation by the empirical density (which is represented by a the positions of a set of “particles”) one obtains an interacting particle ﬂow which converges asymptotically to an empirical approximation of the desired optimal variational density.
Revised: Using an approximation by the empirical density (which is represented by a the positions of a set of “particles”) one obtains an interacting particle ﬂow which converges asymptotically to an empirical approximation of the desired optimal variational density.

Original: A review of related work in VI and ParVI is given in Appendix A.
Revised: A review of related work in VI and ParVI is given in Appendix A.

Original: 2.2. Gaussian density parametrization
Revised: 2.2. Gaussian density parametrization

Original: We want to optimize the parameters θ of a Gaussian variational distribution q θ ( x ) to closely approximate the target density p ( x ) ∝ exp( − ϕ ( x )), where x ∈ R D .
Revised: We want to optimize the parameters θ of a Gaussian variational distribution q θ ( x ) to closely approximate the target density p ( x ) ∝ exp( − ϕ ( x )), where x ∈ R D .

Original: For example, for Bayesian inference, ϕ ( x ) = − log( p ( y | x )) − log( p ( x )).
Revised: For example, for Bayesian inference, ϕ ( x ) = − log( p ( y | x )) − log( p ( x )).

Original: To this end, we minimize the free energy F described in Equation (1).
Revised: To this end, we minimize the free energy F described in Equation (1).

Original: Gradient based minimization is facilitated by the so–called reparametrization trick .
Revised: Gradient based minimization is facilitated by the so–called reparametrization trick .

Original: This replaces the expectation over the parameter dependent variational density q θ by an expectation over a ﬁxed density q 0 instead.
Revised: This replaces the expectation over the parameter dependent variational density q θ by an expectation over a ﬁxed density q 0 instead.

Original: In this way, unwanted derivatives of the type ∇ θ q θ ( x ) are avoided.
Revised: In this way, unwanted derivatives of the type ∇ θ q θ ( x ) are avoided.

Original: Any D dimensional random variable x ∼ q θ ( x ) can be constructed by a linear parametrization of a D -dimensional Gaussian random variable x 0 ∼ q 0 = N ( m
Revised: Any D dimensional random variable x ∼ q θ ( x ) can be constructed by a linear parametrization of a D -dimensional Gaussian random variable x 0 ∼ q 0 = N ( m

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where Γ ∈ R D × D and m ∈ R D are the variational parameters.
Revised: where Γ ∈ R D × D and m ∈ R D are the variational parameters.

Original: The free energy is easily obtained as a function of the variational parameters:
Revised: The free energy is easily obtained as a function of the variational parameters:

Original: [Equation]
Revised: [Equation]

Original: To minimize F (Γ , m ) with respect to its parameters, we deﬁne a gradient ﬂow Γ( t ), m ( t ) for t ≥ 0 in parameter space.
Revised: To minimize F (Γ , m ) with respect to its parameters, we deﬁne a gradient ﬂow Γ( t ), m ( t ) for t ≥ 0 in parameter space.

Original: For the matrix valued parameter Γ, we apply a natural gradient learning approach.
Revised: For the matrix valued parameter Γ, we apply a natural gradient learning approach.

Original: This was introduced by S. Amari (1998, 7.6) as an eﬃcient technique for learning the mixing matrix in models of blind source separation.
Revised: This was introduced by S. Amari (1998, 7.6) as an eﬃcient technique for learning the mixing matrix in models of blind source separation.

Original: We give a more detailed description on natural gradient theory in Appendix B. A combination of the natural gradient for Γ and an ordinary gradient for m leads to the following ﬂow:
Revised: We give a more detailed description on natural gradient theory in Appendix B. A combination of the natural gradient for Γ and an ordinary gradient for m leads to the following ﬂow:

Original: [Equation]
Revised: [Equation]

Original: where we have omitted the time dependency of the parameters.
Revised: where we have omitted the time dependency of the parameters.

Original: One can easily verify that the ﬂow guarantees a decrease of F over time.
Revised: One can easily verify that the ﬂow guarantees a decrease of F over time.

Original: The explicit form of the ﬂows is given by
Revised: The explicit form of the ﬂows is given by

Original: [Equation]
Revised: [Equation]

Original: where A = (cid:0) E q 0 (cid:2) ∇ x ϕ ( x )( x − m ) (cid:62) (cid:3) − I (cid:1) .
Revised: where A = (cid:0) E q 0 (cid:2) ∇ x ϕ ( x )( x − m ) (cid:62) (cid:3) − I (cid:1) .

Original: Using Stein’s lemma (Ingersoll, 1987), we can show that the ﬁxed-point solution of Equations (5) equal the conditions for the optimal variational Gaussian distribution solution given in Opper and Archambeau (2009).
Revised: Using Stein’s lemma (Ingersoll, 1987), we can show that the ﬁxed-point solution of Equations (5) equal the conditions for the optimal variational Gaussian distribution solution given in Opper and Archambeau (2009).

Original: A standard algorithmic approach to deal with the ﬂow equations would be to discretize
Revised: A standard algorithmic approach to deal with the ﬂow equations would be to discretize

Original: Equations (5) in time, and estimate expectations by drawing independent samples from the ﬁxed Gaussian q 0 at each time step.
Revised: Equations (5) in time, and estimate expectations by drawing independent samples from the ﬁxed Gaussian q 0 at each time step.

Original: A stochastic gradient version of this algorithm would combine a small number of samples with a decreasing learning rate, we refer later to this approach as GDP and describe all algorithms steps in Algorithm 2 in Appendix D.
Revised: A stochastic gradient version of this algorithm would combine a small number of samples with a decreasing learning rate, we refer later to this approach as GDP and describe all algorithms steps in Algorithm 2 in Appendix D.

Original: Remarkably, this scheme is diﬀerent from previous algorithms for Gaussian variational inference (e.g., see Opper and Archambeau (2009);
Revised: Remarkably, this scheme is diﬀerent from previous algorithms for Gaussian variational inference (e.g., see Opper and Archambeau (2009);

Original: Titsias and L´azaro-Gredilla (2014))
Revised: Titsias and L´azaro-Gredilla (2014))

Original: as no matrix inversions or second derivatives of the function ϕ are required.
Revised: as no matrix inversions or second derivatives of the function ϕ are required.

Original: We will next introduce a particle based algorithm as an approximation of the ﬂow instead.
Revised: We will next introduce a particle based algorithm as an approximation of the ﬂow instead.

Original: 2.3. Particle dynamics
Revised: 2.3. Particle dynamics

Original: The dynamics of the parameters m and Γ induces a corresponding ﬂow for the random variable x via (2).
Revised: The dynamics of the parameters m and Γ induces a corresponding ﬂow for the random variable x via (2).

Original: Applying Equations (5) on Equation (2) we obtain:
Revised: Applying Equations (5) on Equation (2) we obtain:

Original: [Equation]
Revised: [Equation]

Original: Note that at each time t , A , m and E q 0 [ ∇ x ϕ ( x )] still depend on the parameters Γ( t ) and m ( t ).
Revised: Note that at each time t , A , m and E q 0 [ ∇ x ϕ ( x )] still depend on the parameters Γ( t ) and m ( t ).

Original: In Appendix C, we also give an alternative approach solely based on linear dynamics to obtain Equation (6), without considering parameters like m and Γ.
Revised: In Appendix C, we also give an alternative approach solely based on linear dynamics to obtain Equation (6), without considering parameters like m and Γ.

Original: This dependency can be removed by undoing the reparametrization and re-introducing expectations over the induced density q tθ of x ( t ) itself.
Revised: This dependency can be removed by undoing the reparametrization and re-introducing expectations over the induced density q tθ of x ( t ) itself.

Original: The main idea of the particle approach is to approximate this
Revised: The main idea of the particle approach is to approximate this

Original: Gaussian density by the empirical distribution ˆ q t computed from N samples x ti , i = 1 , . . .
Revised: Gaussian density by the empirical distribution ˆ q t computed from N samples x ti , i = 1 , . . .

Original: , N .
Revised: , N .

Original: These are sampled initially from from the density q 0 at time t = 0 and are then propagated using the discretized dynamics (6):
Revised: These are sampled initially from from the density q 0 at time t = 0 and are then propagated using the discretized dynamics (6):

Original: [Equation]
Revised: [Equation]

Original: where η t 1 and η t 2 are learning rates to avoid unstable steps and m t = 1 N (cid:80) N i
Revised: where η t 1 and η t 2 are learning rates to avoid unstable steps and m t = 1 N (cid:80) N i

Original: =1 x t i .
Revised: =1 x t i .

Original: We give further comments on the use of diﬀerent optimization schemes in Section 3.
Revised: We give further comments on the use of diﬀerent optimization schemes in Section 3.

Original: It is easy to see that the particle dynamics also minimizes the empirical approximation of the free energy (3) obtained by replacing the expectation over q 0 with its empirical counterpart ˆ q 0 .
Revised: It is easy to see that the particle dynamics also minimizes the empirical approximation of the free energy (3) obtained by replacing the expectation over q 0 with its empirical counterpart ˆ q 0 .

Original: 2.4. Algorithm
Revised: 2.4. Algorithm

Original: The algorithm we propose is to sample N particles { x 01 , . . .
Revised: The algorithm we propose is to sample N particles { x 01 , . . .

Original: , x 0 N } where x 0 i ∈
Revised: , x 0 N } where x 0 i ∈

Original: R D from q(which can be centered around the MAP), and iteratively optimize their positions using
Revised: R D from q(which can be centered around the MAP), and iteratively optimize their positions using

Original: Equation (7).
Revised: Equation (7).

Original: Once convergence is reached, i.e., d F dt = 0, we can easily work with the obtained empirical distribution ˆ q ( x ) =
Revised: Once convergence is reached, i.e., d F dt = 0, we can easily work with the obtained empirical distribution ˆ q ( x ) =

Original: 1 N (cid:80) Ni =

Revised: 1 N (cid:80) Ni =

Original: δ x,x i or alternatively the Gaussian density it represents, namely q ( x ) =
Revised: δ x,x i or alternatively the Gaussian density it represents, namely q ( x ) =

Original: N ( m, C ), where m = 1 N (cid:80) Ni
Revised: N ( m, C ), where m = 1 N (cid:80) Ni

Original: =1 x i and C = 1 N (cid:80) Ni
Revised: =1 x i and C = 1 N (cid:80) Ni

Original: =1 ( x i − m )( x i − m ) (cid:62) .
Revised: =1 ( x i − m )( x i − m ) (cid:62) .

Original: To draw samples from q , the empirical covariance C does not need to be computed, as we can obtain new samples by computing:
Revised: To draw samples from q , the empirical covariance C does not need to be computed, as we can obtain new samples by computing:

Original: [Equation]
Revised: [Equation]

Original: where ξ i are i.i.d.
Revised: where ξ i are i.i.d.

Original: normal variables: ξ i ∼ N (0 , 1).
Revised: normal variables: ξ i ∼ N (0 , 1).

Original: All the inference steps are summarized in Algorithm 1 in Appendix D.
Revised: All the inference steps are summarized in Algorithm 1 in Appendix D.

Original: Our algorithm does not require computing second order derivatives and, more crucially, it also does not involve matrix inversions (unlike GVA) and thus scales well with the dimensionality of the problem.
Revised: Our algorithm does not require computing second order derivatives and, more crucially, it also does not involve matrix inversions (unlike GVA) and thus scales well with the dimensionality of the problem.

Original: The complexity of the method compares favorably to GVA.
Revised: The complexity of the method compares favorably to GVA.

Original: In GVA we would need to perform matrix inversions of complexity
Revised: In GVA we would need to perform matrix inversions of complexity

Original: O
Revised: O

Original: ( D 3 ), which we avoid.
Revised: ( D 3 ), which we avoid.

Original: The computational bottleneck now lies in the gradients g ( x i ), adding a storing complexity of O ( ND ), and in the matrix multiplication of A ( x − m ), which has a total computational complexity of O ( N 2 D ).
Revised: The computational bottleneck now lies in the gradients g ( x i ), adding a storing complexity of O ( ND ), and in the matrix multiplication of A ( x − m ), which has a total computational complexity of O ( N 2 D ).

Original: The direct consequence is that, for fewer particles than dimensions ( N ≤ D +1), we improve speed through a low-rank approximation of the covariance matrix.
Revised: The direct consequence is that, for fewer particles than dimensions ( N ≤ D +1), we improve speed through a low-rank approximation of the covariance matrix.

Original: Dynamics and ﬁxed points for Gaussian targets
Revised: Dynamics and ﬁxed points for Gaussian targets

Original: We can obtain some exact theoretical results for the dynamics and the ﬁxed points of our algorithm when the target is a multivariate Gaussian density.
Revised: We can obtain some exact theoretical results for the dynamics and the ﬁxed points of our algorithm when the target is a multivariate Gaussian density.

Original: While such targets may seem like a trivial application, our analysis might still give some intuition about the performance for more complicated densities.
Revised: While such targets may seem like a trivial application, our analysis might still give some intuition about the performance for more complicated densities.

Original: Theorem 

Revised: Theorem

Original: If the target density p ( x ) is a D -dimensional multivariate Gaussian, only D + particles are needed for Algorithm 1 to converge to the exact target parameters.
Revised: If the target density p ( x ) is a D -dimensional multivariate Gaussian, only D + particles are needed for Algorithm 1 to converge to the exact target parameters.

Original: Proof: The proof is given in Appendix E.
Revised: Proof: The proof is given in Appendix E.

Original: Theorem 2 For a target p ( x ) =
Revised: Theorem 2 For a target p ( x ) =

Original: N ( x | µ, Λ − 1 ) , where x ∈ R D , and N ≥ D + 1 particles, the continuous time limit of Algorithm 1 will converge linearly for both the mean and the trace of the precision matrix:
Revised: N ( x | µ, Λ − 1 ) , where x ∈ R D , and N ≥ D + 1 particles, the continuous time limit of Algorithm 1 will converge linearly for both the mean and the trace of the precision matrix:

Original: [Equation]
Revised: [Equation]

Original: where m t and C t are the empirical mean and covariance matrix at time t and exp( − Λ t ) is the matrix exponential.
Revised: where m t and C t are the empirical mean and covariance matrix at time t and exp( − Λ t ) is the matrix exponential.

Original: Proof: The proof is given in Appendix F.
Revised: Proof: The proof is given in Appendix F.

Original: Our result shows that convergence of the mean m t depends on the eigenvalues of Λ.
Revised: Our result shows that convergence of the mean m t depends on the eigenvalues of Λ.

Original: From our partial result for the covariance C t we might conjecture that convergence of C t could be (asymptotically) independent of the target parameters.
Revised: From our partial result for the covariance C t we might conjecture that convergence of C t could be (asymptotically) independent of the target parameters.

Original: We conﬁrm this behaviour
Revised: We conﬁrm this behaviour

Original: [Equation]
Revised: [Equation]

Original: Isotropic covariance
Revised: Isotropic covariance

Original: [Equation]
Revised: [Equation]

Original: where U is unitary and
Revised: where U is unitary and

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: for Gaussian targets in the experiments in Section 3.
Revised: for Gaussian targets in the experiments in Section 3.

Original: We also show that good approximation can be obtained for a moderate number of particles.
Revised: We also show that good approximation can be obtained for a moderate number of particles.

Original: If the number of particles is in fact equal or smaller than the dimension, i.e. N < D + 1, we can show the following result for the ﬁxed points:
Revised: If the number of particles is in fact equal or smaller than the dimension, i.e. N < D + 1, we can show the following result for the ﬁxed points:

Original: Theorem 3 Given a D -dimensional multivariate Gaussian target density p ( x ) =
Revised: Theorem 3 Given a D -dimensional multivariate Gaussian target density p ( x ) =

Original: N ( x | µ, Σ) , using Algorithm 1 with N < D + 1 particles, the empirical mean converges to the exact mean µ .
Revised: N ( x | µ, Σ) , using Algorithm 1 with N < D + 1 particles, the empirical mean converges to the exact mean µ .

Original: The N − 1 non-zero eigenvalues of C t converge to a subset of the spectrum of the target covariance Σ .
Revised: The N − 1 non-zero eigenvalues of C t converge to a subset of the spectrum of the target covariance Σ .

Original: Proof: The proof is given in Appendix G. Supported by simulations, we conjecture that we recover in fact the largest eigenvalues of Σ in this way.
Revised: Proof: The proof is given in Appendix G. Supported by simulations, we conjecture that we recover in fact the largest eigenvalues of Σ in this way.

Original: This results suggests that it makes sense to apply our algorithm to high–dimensional problems even when the number of particles is not large.
Revised: This results suggests that it makes sense to apply our algorithm to high–dimensional problems even when the number of particles is not large.

Original: If the target density has signiﬁcant support in a low dimensional sub–manifold only, we might still get a reasonable approximation.
Revised: If the target density has signiﬁcant support in a low dimensional sub–manifold only, we might still get a reasonable approximation.

Original: 3. Experiments
Revised: 3. Experiments

Original: We now demonstrate empirically the beneﬁts of GPF.
Revised: We now demonstrate empirically the beneﬁts of GPF.

Original: First, we explore how the diﬀerent algorithm hyperparameters aﬀect inference.
Revised: First, we explore how the diﬀerent algorithm hyperparameters aﬀect inference.

Original: Second, we compare the ability of GPF to recover the target posterior distribution with popular VI methods, including SVGD ands
Revised: Second, we compare the ability of GPF to recover the target posterior distribution with popular VI methods, including SVGD ands

Original: GVA.
Revised: GVA.

Original: In particular, we work fully-controlled synthetic examples based on Gaussian target distributions with an increasing number of dimensions as well as highly non-
Revised: In particular, we work fully-controlled synthetic examples based on Gaussian target distributions with an increasing number of dimensions as well as highly non-

Original: Gaussian posteriors.
Revised: Gaussian posteriors.

Original: We evaluate the ability of GPF to recover the ground truth as the number of particles increases and compare it to competing VI approaches.
Revised: We evaluate the ability of GPF to recover the ground truth as the number of particles increases and compare it to competing VI approaches.

Original: We use standard gradient descent with η = 0 .
Revised: We use standard gradient descent with η = 0 .

Original: 1 as well as a clipping on the gradient norm with a threshold (cid:15)
Revised: 1 as well as a clipping on the gradient norm with a threshold (cid:15)

Original: =
Revised: =

Original: in all our experiments.
Revised: in all our experiments.

Original: We run each experiment for 2000 iterations and aggregate results over 10 independent runs.
Revised: We run each experiment for 2000 iterations and aggregate results over 10 independent runs.

Original: We also evaluate the eﬃciency of our algorithm on Gaussian
Revised: We also evaluate the eﬃciency of our algorithm on Gaussian

Original: Processes (GP) s and Bayesian Neural Networks (BNN) s in Appendix H.
Revised: Processes (GP) s and Bayesian Neural Networks (BNN) s in Appendix H.

Original: Gaussian Targets
Revised: Gaussian Targets

Original: Consider a D -dimensional Gaussian target.
Revised: Consider a D -dimensional Gaussian target.

Original: The mean is sampled from a normal Gaussian µ ∼ N (0 , I D ) and we consider two diﬀerent cases for the covariance: isotropic Σ
Revised: The mean is sampled from a normal Gaussian µ ∼ N (0 , I D ) and we consider two diﬀerent cases for the covariance: isotropic Σ

Original: = I D and full-rank covariance Σ
Revised: = I D and full-rank covariance Σ

Original: = U Λ U (cid:62) , where U is a unitary matrix and Λ is a diagonal matrix where Λ ii = 10 3 i − 1 D − 1 .
Revised: = U Λ U (cid:62) , where U is a unitary matrix and Λ is a diagonal matrix where Λ ii = 10 3 i − 1 D − 1 .

Original: We compare GPF and GDP to two state-of-the art methods described in Appendix A, namely SVGD and GVA.
Revised: We compare GPF and GDP to two state-of-the art methods described in Appendix A, namely SVGD and GVA.

Original: We vary the number of dimensions D and set the number of particles or alternatively the number of samples for
Revised: We vary the number of dimensions D and set the number of particles or alternatively the number of samples for

Original: GVA to D + 1.
Revised: GVA to D + 1.

Original: Figure 2.4 reports the L 2 norm on the diﬀerence of mean and covariance with the true posterior, and the inference time required by the competing methods.
Revised: Figure 2.4 reports the L 2 norm on the diﬀerence of mean and covariance with the true posterior, and the inference time required by the competing methods.

Original: GPF recovers the true mean and covariance at a fraction of the time required by SVGD and
Revised: GPF recovers the true mean and covariance at a fraction of the time required by SVGD and

Original: GVA.
Revised: GVA.

Original: The gap is more pronounced as the target dimensionality increases.
Revised: The gap is more pronounced as the target dimensionality increases.

Original: Non-Gaussian target distributions and optimizers
Revised: Non-Gaussian target distributions and optimizers

Original: We investigate non-Gaussian and multi-modal target distributions.
Revised: We investigate non-Gaussian and multi-modal target distributions.

Original: We compare diﬀerent optimizers on the 2D banana distribution: p ( x ) ∝ exp (cid:0) − 0 .
Revised: We compare diﬀerent optimizers on the 2D banana distribution: p ( x ) ∝ exp (cid:0) − 0 .

Original: 01 x 21 + 0 .
Revised: 01 x 21 + 0 .

Original: 1( x 2 + 0 .
Revised: 1( x 2 + 0 .

Original: 1 x 21 − 10) 2 ) (cid:1) .
Revised: 1 x 21 − 10) 2 ) (cid:1) .

Original: We vary the number of particles used for inference in { 3 , 5 , 10 , 20 , 50 } and compare GPF with the standard GVA approach.
Revised: We vary the number of particles used for inference in { 3 , 5 , 10 , 20 , 50 } and compare GPF with the standard GVA approach.

Original: We also show the impact of replacing SGD with Adam (Kingma and Ba, 2014) for 50 particles.
Revised: We also show the impact of replacing SGD with Adam (Kingma and Ba, 2014) for 50 particles.

Original: The results are shown in Figure 3.
Revised: The results are shown in Figure 3.

Original: As expected, increasing the number of particles makes the distribution obtained via GPF increasingly closer to the one obtained by the standard GVA at much lower computational cost.
Revised: As expected, increasing the number of particles makes the distribution obtained via GPF increasingly closer to the one obtained by the standard GVA at much lower computational cost.

Original: Using a momentum based optimizer such as Adam breaks the Gaussian distribution and particles concentrate on two modes.
Revised: Using a momentum based optimizer such as Adam breaks the Gaussian distribution and particles concentrate on two modes.

Original: This is caused by momentum-based optimizers breaking the linearity of Equation (6), which could studied as a desirable property to ﬁt multi-modal distributions in future work.
Revised: This is caused by momentum-based optimizers breaking the linearity of Equation (6), which could studied as a desirable property to ﬁt multi-modal distributions in future work.

Original: 4. Conclusion
Revised: 4. Conclusion

Original: We introduced GPF and GDP, general-purpose, scalable, and theoretically-grounded approaches to perform VI.
Revised: We introduced GPF and GDP, general-purpose, scalable, and theoretically-grounded approaches to perform VI.

Original: The key contribution is to leverage deterministic dynamics to perform tractable GVA in high dimensions, making it applicable to a broad set of challenging, real-world high-dimensional problems.
Revised: The key contribution is to leverage deterministic dynamics to perform tractable GVA in high dimensions, making it applicable to a broad set of challenging, real-world high-dimensional problems.

Original: We also derived theoretical performance guarantees for Gaussian targets.
Revised: We also derived theoretical performance guarantees for Gaussian targets.

Original: We studied the methods on synthetic data as well as more challenging problems.
Revised: We studied the methods on synthetic data as well as more challenging problems.

Original: Investigations are still needed to evaluate the eﬀect of the empirical estimate for the free energy, especially in the regime of a small number of particles.
Revised: Investigations are still needed to evaluate the eﬀect of the empirical estimate for the free energy, especially in the regime of a small number of particles.

Original: Another promising direction is the eﬀect of momentum-based optimizers to obtain multi-modal variational distributions.
Revised: Another promising direction is the eﬀect of momentum-based optimizers to obtain multi-modal variational distributions.

Original: Appendix A. Related Work
Revised: Appendix A. Related Work

Original: A diverse set of approaches to perform VI have been developed in the literature, which we review here.
Revised: A diverse set of approaches to perform VI have been developed in the literature, which we review here.

Original: High-dimensional VI
Revised: High-dimensional VI

Original: The most popular approach to solve high-dimensional problems via variational inference is (structured) Mean-Field (MF) (Hinton and van Camp, 1993;
Revised: The most popular approach to solve high-dimensional problems via variational inference is (structured) Mean-Field (MF) (Hinton and van Camp, 1993;

Original: Blei et al., 2017).
Revised: Blei et al., 2017).

Original: By imposing independence between variables in the variational distribution, the computational complexity decreases at the cost of the approximation quality.
Revised: By imposing independence between variables in the variational distribution, the computational complexity decreases at the cost of the approximation quality.

Original: For
Revised: For

Original: Gaussian distributions the most common approach is to leverage the Gaussian Variational
Revised: Gaussian distributions the most common approach is to leverage the Gaussian Variational

Original: Approximation (GVA) (Opper and Archambeau, 2009), where the expectation gradients given the variational parameters can be obtained directly with a Monte-Carlo estimation.
Revised: Approximation (GVA) (Opper and Archambeau, 2009), where the expectation gradients given the variational parameters can be obtained directly with a Monte-Carlo estimation.

Original: More recently Titsias and L´azaro-Gredilla (2014) avoid the need for integrals by proceeding to stochastic updates based on the idea of Stochastic Gradient Descent (SGD) .
Revised: More recently Titsias and L´azaro-Gredilla (2014) avoid the need for integrals by proceeding to stochastic updates based on the idea of Stochastic Gradient Descent (SGD) .

Original: The most generic technique is Black-Box Variational Inference (Ranganath et al., 2014; Blei et al., 2017), which is based on the reparametrization trick .
Revised: The most generic technique is Black-Box Variational Inference (Ranganath et al., 2014; Blei et al., 2017), which is based on the reparametrization trick .

Original: This avoids unwanted dependencies of the sampling distribution on parameters and yields unbiased estimates of parameter gradients.
Revised: This avoids unwanted dependencies of the sampling distribution on parameters and yields unbiased estimates of parameter gradients.

Original: However, it is nontrivial to apply this idea to a full rank D -dimensional multivariate variational Gaussian density.
Revised: However, it is nontrivial to apply this idea to a full rank D -dimensional multivariate variational Gaussian density.

Original: One possibility would be to resort to Cholesky factorizations of the covariance matrix in the parametrisation.
Revised: One possibility would be to resort to Cholesky factorizations of the covariance matrix in the parametrisation.

Original: This would scale with a prohibitive O
Revised: This would scale with a prohibitive O

Original: ( D 3 ) complexity.
Revised: ( D 3 ) complexity.

Original: In addition, eﬃcient updates based on natural gradient descent methods (Khan and Nielsen, 2018; Lin et al., 2019) require extra second derivatives of log–posteriors.
Revised: In addition, eﬃcient updates based on natural gradient descent methods (Khan and Nielsen, 2018; Lin et al., 2019) require extra second derivatives of log–posteriors.

Original: These derivatives may not be always available in a black-box scenario.
Revised: These derivatives may not be always available in a black-box scenario.

Original: Hence, extra approximations using outer products of gradients are used instead.
Revised: Hence, extra approximations using outer products of gradients are used instead.

Original: Particle-Based VI
Revised: Particle-Based VI

Original: Stochastic gradient descent methods compute expectations (and gradients) with Monte Carlo samples from the current approximation of the variational density independently at each time step.
Revised: Stochastic gradient descent methods compute expectations (and gradients) with Monte Carlo samples from the current approximation of the variational density independently at each time step.

Original: Particle based methods for variational inference draw samples once at the beginning of the algorithm instead.
Revised: Particle based methods for variational inference draw samples once at the beginning of the algorithm instead.

Original: They iteratively construct a transformation on an initial random variable (with a simple tractable density) leading to the minimization of the variational free energy.
Revised: They iteratively construct a transformation on an initial random variable (with a simple tractable density) leading to the minimization of the variational free energy.

Original: The iterative approach induces a deterministic temporal ﬂow of random variables which depends on the current density of the variable itself.
Revised: The iterative approach induces a deterministic temporal ﬂow of random variables which depends on the current density of the variable itself.

Original: Using an approximation by the empirical density (which is represented by a the positions of a set of “particles”) one obtains an interacting particle ﬂow which converges asymptotically to an empirical approximation of the desired optimal variational density.
Revised: Using an approximation by the empirical density (which is represented by a the positions of a set of “particles”) one obtains an interacting particle ﬂow which converges asymptotically to an empirical approximation of the desired optimal variational density.

Original: The most popular approach is SVGD (Liu and Wang, 2016), which ﬁnds a nonparametric transport map transforming a collection of particles into samples from the target distribution based on the kernelized Stein discrepancy (Liu et al., 2016).
Revised: The most popular approach is SVGD (Liu and Wang, 2016), which ﬁnds a nonparametric transport map transforming a collection of particles into samples from the target distribution based on the kernelized Stein discrepancy (Liu et al., 2016).

Original: SVGD has the advantage of not being restricted to a parametric form of the variational distribution.
Revised: SVGD has the advantage of not being restricted to a parametric form of the variational distribution.

Original: On the other hand, useful features such as an explicit expression for the variational free energy or structured approximations, such as a mean-ﬁeld method, are not yet available.
Revised: On the other hand, useful features such as an explicit expression for the variational free energy or structured approximations, such as a mean-ﬁeld method, are not yet available.

Original: A mean ﬁeld approximation could have a regularising eﬀect for ﬁnite sample sizes in high dimensions.
Revised: A mean ﬁeld approximation could have a regularising eﬀect for ﬁnite sample sizes in high dimensions.

Original: Other limitations of SVGD are the quadratic scaling with the number of samples and the approximation of the kernelized Stein discrepancy.
Revised: Other limitations of SVGD are the quadratic scaling with the number of samples and the approximation of the kernelized Stein discrepancy.

Original: This is known to not perform well in high dimensions (Zhuo et al., 2018).
Revised: This is known to not perform well in high dimensions (Zhuo et al., 2018).

Original: Non-parametric Variational Inference presents a diﬀerent approach where each particle represents a Gaussian (similarly to Kernel Density Estimation in Fix and Hodges (1951) and Cacoullos (1964) with a Gaussian kernel) and the KL-divergence between the mixture of Gaussians and the target posterior is minimized (Gershman et al., 2012).
Revised: Non-parametric Variational Inference presents a diﬀerent approach where each particle represents a Gaussian (similarly to Kernel Density Estimation in Fix and Hodges (1951) and Cacoullos (1964) with a Gaussian kernel) and the KL-divergence between the mixture of Gaussians and the target posterior is minimized (Gershman et al., 2012).

Original: However, due to the complicated behavior of Gaussian densities in high-dimensions, this method does not scale well.
Revised: However, due to the complicated behavior of Gaussian densities in high-dimensions, this method does not scale well.

Original: Variational Particle Approximation (Saeedi et al., 2017) also use particles to represent a variational distribution, but the method is only applicable to discrete variables.
Revised: Variational Particle Approximation (Saeedi et al., 2017) also use particles to represent a variational distribution, but the method is only applicable to discrete variables.

Original: Related approaches
Revised: Related approaches

Original: The closest approach to our proposed method is Ensemble Kalman
Revised: The closest approach to our proposed method is Ensemble Kalman

Original: Filter (Evensen, 1994), where a Gaussian distribution is repeatedly approximated by a set of particles, iteratively moved following the Kalman ﬁlter updates.
Revised: Filter (Evensen, 1994), where a Gaussian distribution is repeatedly approximated by a set of particles, iteratively moved following the Kalman ﬁlter updates.

Original: However, this method is typically applied to an online context, which poses problems that do not hold in our variational setting (Ehrendorfer, 2007).
Revised: However, this method is typically applied to an online context, which poses problems that do not hold in our variational setting (Ehrendorfer, 2007).

Original: A related method recently introduced in the context of
Revised: A related method recently introduced in the context of

Original: BNN models is Stochastic Weight Averaging - Gaussian (SWAG) (Maddox et al., 2019), in which a set of particles obtained via stochastic gradient descent represent a low-rank
Revised: BNN models is Stochastic Weight Averaging - Gaussian (SWAG) (Maddox et al., 2019), in which a set of particles obtained via stochastic gradient descent represent a low-rank

Original: Gaussian distribution, approximating the true posterior with a prior produced by the network’s regularization.
Revised: Gaussian distribution, approximating the true posterior with a prior produced by the network’s regularization.

Original: While easy to implement, SWAG does not allow for incorporating an explicit prior.
Revised: While easy to implement, SWAG does not allow for incorporating an explicit prior.

Original: Finally, we mention Normalizing Flows (NF) (Rezende and Mohamed, 2015), where bijections are sequentially stacked on top of a ﬁxed initial distribution.
Revised: Finally, we mention Normalizing Flows (NF) (Rezende and Mohamed, 2015), where bijections are sequentially stacked on top of a ﬁxed initial distribution.

Original: The repeated transformations on an initial distribution can be viewed as a discrete ﬂow.
Revised: The repeated transformations on an initial distribution can be viewed as a discrete ﬂow.

Original: This is orthogonal to our approach and could be combined with GPF by applying NF on the variational distribution.
Revised: This is orthogonal to our approach and could be combined with GPF by applying NF on the variational distribution.

Original: For instance, this makes it possible to work with diﬀerent domains than R .
Revised: For instance, this makes it possible to work with diﬀerent domains than R .

Original: Appendix B. Natural gradient for matrix parameter Γ
Revised: Appendix B. Natural gradient for matrix parameter Γ

Original: The natural gradient is deﬁned as the parameter change d Γ =
Revised: The natural gradient is deﬁned as the parameter change d Γ =

Original: Γ( t + dt ) − Γ( t ) over a small time interval dt which yields the direction of steepest descent of the free energy.
Revised: Γ( t + dt ) − Γ( t ) over a small time interval dt which yields the direction of steepest descent of the free energy.

Original: As an extra condition one keeps the length of d Γ (measured by a ’natural’ metric which has speciﬁc invariance properties) ﬁxed.
Revised: As an extra condition one keeps the length of d Γ (measured by a ’natural’ metric which has speciﬁc invariance properties) ﬁxed.

Original: The metric is deﬁned by an inner product (the squared length) (cid:104) d Γ , d Γ (cid:105) Γ in the tangent space of small deviations d Γ from the matrix Γ. Hence, d Γ is found by minimising F (Γ( t ) + d Γ , m ) (for small d Γ) under the condition that (cid:104) d Γ , d Γ (cid:105) Γ( t ) is ﬁxed.
Revised: The metric is deﬁned by an inner product (the squared length) (cid:104) d Γ , d Γ (cid:105) Γ in the tangent space of small deviations d Γ from the matrix Γ. Hence, d Γ is found by minimising F (Γ( t ) + d Γ , m ) (for small d Γ) under the condition that (cid:104) d Γ , d Γ (cid:105) Γ( t ) is ﬁxed.

Original: Following Amari (1998, 7.6) a natural metric in the space of symmetric nonsingular matrices can be deﬁned as
Revised: Following Amari (1998, 7.6) a natural metric in the space of symmetric nonsingular matrices can be deﬁned as

Original: [Equation]
Revised: [Equation]

Original: This metric is invariant against multiplications of Γ and d Γ by matrices Y , i.e. (cid:104)
Revised: This metric is invariant against multiplications of Γ and d Γ by matrices Y , i.e. (cid:104)

Original: d Γ , d Γ (cid:105) Γ = (cid:104)
Revised: d Γ , d Γ (cid:105) Γ = (cid:104)

Original: d Γ Y, d Γ Y (cid:105) Γ Y and reduces to the Euclidian metric at the unit matrix Γ
Revised: d Γ Y, d Γ Y (cid:105) Γ Y and reduces to the Euclidian metric at the unit matrix Γ

Original: = I .
Revised: = I .

Original: The direction of the natural gradient is obtained by expanding the free energy for small d Γ and introducing a Lagrange–multiplier λ for the constraint.
Revised: The direction of the natural gradient is obtained by expanding the free energy for small d Γ and introducing a Lagrange–multiplier λ for the constraint.

Original: One ends up with the quadratic form
Revised: One ends up with the quadratic form

Original: [Equation]
Revised: [Equation]

Original: to be minimised by d Γ. By taking the derivative with respect to d Γ one ﬁnds that the direction of d Γ agrees with the right equation of the ﬂow (4).
Revised: to be minimised by d Γ. By taking the derivative with respect to d Γ one ﬁnds that the direction of d Γ agrees with the right equation of the ﬂow (4).

Original: Appendix C. Free Energy as a function of particles
Revised: Appendix C. Free Energy as a function of particles

Original: In Equation (3) we have the free energy F as a function of parameters Γ and m .
Revised: In Equation (3) we have the free energy F as a function of parameters Γ and m .

Original: Here instead, we only consider the random variable x ∼ q ( x ), as well as its mean m =
Revised: Here instead, we only consider the random variable x ∼ q ( x ), as well as its mean m =

Original: E q [ x ] and its covariance C =
Revised: E q [ x ] and its covariance C =

Original: E q (cid:2) ( x − m )( x
Revised: E q (cid:2) ( x − m )( x

Original: [Equation]
Revised: [Equation]

Original: m ) (cid:62) (cid:3) .
Revised: m ) (cid:62) (cid:3) .

Original: The free energy as a function of x is given by
Revised: The free energy as a function of x is given by

Original: [Equation]
Revised: [Equation]

Original: Similarly to the derivations in Section 2, q ( x ) is either a Gaussian distribution q ( x ) ∼
Revised: Similarly to the derivations in Section 2, q ( x ) is either a Gaussian distribution q ( x ) ∼

Original: N ( m, C ) or an empirical distribution ˆ q ( x ) = 1 N (cid:80) Ni =

Revised: N ( m, C ) or an empirical distribution ˆ q ( x ) = 1 N (cid:80) Ni =

Original: δ x,x i with a ﬁnite number of particles.
Revised: δ x,x i with a ﬁnite number of particles.

Original: We now consider time dynamics on x , i.e. x = x ( t ).
Revised: We now consider time dynamics on x , i.e. x = x ( t ).

Original: The induced dynamics of F are:
Revised: The induced dynamics of F are:

Original: [Equation]
Revised: [Equation]

Original: For notation simplicity we deﬁne g ( x ) = ∇ x ϕ ( x ) and ˙ x = dxdt (similarly ˙ m = dmdt ).
Revised: For notation simplicity we deﬁne g ( x ) = ∇ x ϕ ( x ) and ˙ x = dxdt (similarly ˙ m = dmdt ).

Original: [Equation]
Revised: [Equation]

Original: where we used the permutation properties of the trace.
Revised: where we used the permutation properties of the trace.

Original: We now study the eﬀect of linear dynamics on x , i.e.: ˙ x
Revised: We now study the eﬀect of linear dynamics on x , i.e.: ˙ x

Original: = b + A ( x
Revised: = b + A ( x

Original: [Equation]
Revised: [Equation]

Original: m ) .
Revised: m ) .

Original: Plugging those dynamics into Equation (9), we get:
Revised: Plugging those dynamics into Equation (9), we get:

Original: [Equation]
Revised: [Equation]

Original: where we ignored the term b (cid:62) C − 1 E q
Revised: where we ignored the term b (cid:62) C − 1 E q

Original: [ x − m ] as it is naturally equal to 0.
Revised: [ x − m ] as it is naturally equal to 0.

Original: We now ﬁnd terms for b and A such that d F dt < 0, i.e. the dynamics will lead to a decrease of the free energy.
Revised: We now ﬁnd terms for b and A such that d F dt < 0, i.e. the dynamics will lead to a decrease of the free energy.

Original: We pick b = − β 1 E q [ g ( x )], where β 1 > 0, and obtain a negative ﬁrst term:
Revised: We pick b = − β 1 E q [ g ( x )], where β 1 > 0, and obtain a negative ﬁrst term:

Original: [Equation]
Revised: [Equation]

Original: For A let’s ﬁrst deﬁne ψ =
Revised: For A let’s ﬁrst deﬁne ψ =

Original: E q (cid:2) g
Revised: E q (cid:2) g

Original: ( x )( x − m ) (cid:62) (cid:3) and rewrite the second and last term of the Equation 10 as:
Revised: ( x )( x − m ) (cid:62) (cid:3) and rewrite the second and last term of the Equation 10 as:

Original: [Equation]
Revised: [Equation]

Original: =tr
Revised: =tr

Original: [Equation]
Revised: [Equation]

Original: A
Revised: A

Original: [Equation]
Revised: [Equation]

Original: Combining both we get: tr (cid:0) A (cid:62) ( ψ − I ) (cid:1) .
Revised: Combining both we get: tr (cid:0) A (cid:62) ( ψ − I ) (cid:1) .

Original: Similarly to the previous step we pick A = − β 2 ( ψ − I ), where β 2 ≥ 0, which leads to another negative term:
Revised: Similarly to the previous step we pick A = − β 2 ( ψ − I ), where β 2 ≥ 0, which leads to another negative term:

Original: [Equation]
Revised: [Equation]

Original: where we use the fact that X (cid:62) X is a positive semi-deﬁnite matrix for any real valued X and that its eigenvalues are all greater or equal to 0.
Revised: where we use the fact that X (cid:62) X is a positive semi-deﬁnite matrix for any real valued X and that its eigenvalues are all greater or equal to 0.

Original: Note that a diﬀerent A could be used as long as the trace of the product stays positive.
Revised: Note that a diﬀerent A could be used as long as the trace of the product stays positive.

Original: Replacing b and A , the free energy dynamics become:
Revised: Replacing b and A , the free energy dynamics become:

Original: [Equation]
Revised: [Equation]

Original: And the variable dynamics are:
Revised: And the variable dynamics are:

Original: [Equation]
Revised: [Equation]

Original: which is equivalent to Equation (6), for β 1 = β 2 = 1.
Revised: which is equivalent to Equation (6), for β 1 = β 2 = 1.

Original: As in Equation (7), when considering a ﬁnite number of samples, we use instead an empirical approximation of the free energy and replace the expectations by empirical estimates.
Revised: As in Equation (7), when considering a ﬁnite number of samples, we use instead an empirical approximation of the free energy and replace the expectations by empirical estimates.

Original: Appendix D. Algorithms Appendix E. Fixed points for a Gaussian model ( N > d )
Revised: Appendix D. Algorithms Appendix E. Fixed points for a Gaussian model ( N > d )

Original: The general ﬁxed-point condition for the dynamics (7) of the position x i for particle i is given by
Revised: The general ﬁxed-point condition for the dynamics (7) of the position x i for particle i is given by

Original: [Equation]
Revised: [Equation]

Original: Input: Number of particles N , initial distribution q 0 , target p ( x ), learning rates η 1 , η
Revised: Input: Number of particles N , initial distribution q 0 , target p ( x ), learning rates η 1 , η

Original: Output: Empirical dist.
Revised: Output: Empirical dist.

Original: q ( x ) = 1 N (cid:80) Ni =

Revised: q ( x ) = 1 N (cid:80) Ni =

Original: δ x,x i
Revised: δ x,x i

Original: Init: Sample N particles from q 0 : { x i } Ni = for j in 1 : T do
Revised: Init: Sample N particles from q 0 : { x i } Ni = for j in 1 : T do

Original: Compute gradients g i = ∇ x log p
Revised: Compute gradients g i = ∇ x log p

Original: x i ) , ∀
Revised: x i ) , ∀

Original: i
Revised: i

Original: Compute means m = 1 N (cid:80) i x i , ¯ g = 1 N (cid:80)
Revised: Compute means m = 1 N (cid:80) i x i , ¯ g = 1 N (cid:80)

Original: i g i
Revised: i g i

Original: Compute matrix A = 1 N (cid:80) i g i ( x i − m ) (cid:62) − I
Revised: Compute matrix A = 1 N (cid:80) i g i ( x i − m ) (cid:62) − I

Original: Update particles x i = − η 1 ¯ g − η 2 A ( x i − m ) , ∀ i end
Revised: Update particles x i = − η 1 ¯ g − η 2 A ( x i − m ) , ∀ i end

Original: Input: Number of samples N , initial parameters m, Γ, target p ( x ), learning rates η 1 , η
Revised: Input: Number of samples N , initial parameters m, Γ, target p ( x ), learning rates η 1 , η

Original: Output: Gaussian distribution q ( x ) = N (cid:0) m, ΓΓ (cid:62) (cid:1) for j in 1 : T do
Revised: Output: Gaussian distribution q ( x ) = N (cid:0) m, ΓΓ (cid:62) (cid:1) for j in 1 : T do

Original: Sample N samples from q 0 : { x 0 i } Ni =
Revised: Sample N samples from q 0 : { x 0 i } Ni =

Original: Project the samples using m and Γ: x i = m + Γ( x 0 i − m 0 )
Revised: Project the samples using m and Γ: x i = m + Γ( x 0 i − m 0 )

Original: Compute gradients g i = ∇ x log p
Revised: Compute gradients g i = ∇ x log p

Original: x i ) , ∀
Revised: x i ) , ∀

Original: i
Revised: i

Original: Compute means m = 1 N (cid:80) i x i , ¯ g = 1 N (cid:80)
Revised: Compute means m = 1 N (cid:80) i x i , ¯ g = 1 N (cid:80)

Original: i g i
Revised: i g i

Original: Compute matrix A = 1 N (cid:80) i g i ( x i − m ) (cid:62) − I
Revised: Compute matrix A = 1 N (cid:80) i g i ( x i − m ) (cid:62) − I

Original: Update parameters m ← m − η 1 ¯ g, Γ ← Γ − η 2 A Γ , ∀ i end for i = 1 , . .
Revised: Update parameters m ← m − η 1 ¯ g, Γ ← Γ − η 2 A Γ , ∀ i end for i = 1 , . .

Original: . , N .
Revised: . , N .

Original: By taking the expectation over all particles we get:
Revised: By taking the expectation over all particles we get:

Original: [Equation]
Revised: [Equation]

Original: (11) where ˆ q is the empirical distributions of particles at the the ﬁxed point.
Revised: (11) where ˆ q is the empirical distributions of particles at the the ﬁxed point.

Original: Note that this result is independent of N , i.e. it is also valid for N = 1.
Revised: Note that this result is independent of N , i.e. it is also valid for N = 1.

Original: For a D -dimensional Gaussian target, i.e. p ( x ) = N ( µ, Σ), we will show that empirical mean and covariance given by the particle algorithm converge to the true mean and covariance matrix of the Gaussian when we use N ≥ D + 1 particles.
Revised: For a D -dimensional Gaussian target, i.e. p ( x ) = N ( µ, Σ), we will show that empirical mean and covariance given by the particle algorithm converge to the true mean and covariance matrix of the Gaussian when we use N ≥ D + 1 particles.

Original: In this setting we have ϕ ( x ) = 1 2 x (cid:62) Σ − 1 x − x (cid:62) Σ − 1 µ .
Revised: In this setting we have ϕ ( x ) = 1 2 x (cid:62) Σ − 1 x − x (cid:62) Σ − 1 µ .

Original: For simpliﬁcation we use natural parameters η 1 = Σ − 1 µ and η 2 = − 12 Σ − 1 and get
Revised: For simpliﬁcation we use natural parameters η 1 = Σ − 1 µ and η 2 = − 12 Σ − 1 and get

Original: [Equation]
Revised: [Equation]

Original: The gradient g ( x ) becomes:
Revised: The gradient g ( x ) becomes:

Original: [Equation]
Revised: [Equation]

Original: At the ﬁxed points we have that dmdt and d Γ dt are equal to 0.
Revised: At the ﬁxed points we have that dmdt and d Γ dt are equal to 0.

Original: For the mean m :
Revised: For the mean m :

Original: [Equation]
Revised: [Equation]

Original: For the matrix Γ:
Revised: For the matrix Γ:

Original: [Equation]
Revised: [Equation]

Original: Where we replaced η 1 by − 2 η 2 m given the previous result and right multiplied by Γ (cid:62) as
Revised: Where we replaced η 1 by − 2 η 2 m given the previous result and right multiplied by Γ (cid:62) as

Original: C =
Revised: C =

Original: ΓΓ (cid:62) .
Revised: ΓΓ (cid:62) .

Original: Now we can only simplify as C = − 12 η − 1= Σ iﬀ C is not singular.
Revised: Now we can only simplify as C = − 12 η − 1= Σ iﬀ C is not singular.

Original: Which is true only if its rank is equal to D , needing D + 1 particles.
Revised: Which is true only if its rank is equal to D , needing D + 1 particles.

Original: Appendix F. Rates of Convergence for Gaussian model F.1. Convergence of the mean
Revised: Appendix F. Rates of Convergence for Gaussian model F.1. Convergence of the mean

Original: Given a Gaussian target p ( x ) =
Revised: Given a Gaussian target p ( x ) =

Original: N ( x | µ, Σ), as described in Appendix E, we have g ( x ) =
Revised: N ( x | µ, Σ), as described in Appendix E, we have g ( x ) =

Original: − 2 η 2 x − η 1 , where η 1 = Σ − 1 µ and η 2 = − 12 Σ − 1 .
Revised: − 2 η 2 x − η 1 , where η 1 = Σ − 1 µ and η 2 = − 12 Σ − 1 .

Original: This transform the of Equations (5) into:
Revised: This transform the of Equations (5) into:

Original: [Equation]
Revised: [Equation]

Original: If now consider the error on m : δm = m − µ we get:
Revised: If now consider the error on m : δm = m − µ we get:

Original: [Equation]
Revised: [Equation]

Original: Therefore we have a linear convergence which is asymptotically governed by the largest eigenvalue of Σ − 1 , i.e. the inverse of the smallest eigenvalue of Σ, λ min .
Revised: Therefore we have a linear convergence which is asymptotically governed by the largest eigenvalue of Σ − 1 , i.e. the inverse of the smallest eigenvalue of Σ, λ min .

Original: F.2. Convergence of the covariance matrix
Revised: F.2. Convergence of the covariance matrix

Original: For the convergence of the covariance we focus once more on the Gaussian case.
Revised: For the convergence of the covariance we focus once more on the Gaussian case.

Original: Let z = x − m , we have that from Equation (6):
Revised: Let z = x − m , we have that from Equation (6):

Original: [Equation]
Revised: [Equation]

Original: where A = E q 0 (cid:2) g ( x ) z (cid:62) (cid:3) − I .
Revised: where A = E q 0 (cid:2) g ( x ) z (cid:62) (cid:3) − I .

Original: This expectation can further simpliﬁed as:
Revised: This expectation can further simpliﬁed as:

Original: [Equation]
Revised: [Equation]

Original: where q ∼ N ( m, C ).
Revised: where q ∼ N ( m, C ).

Original: Hence we have the exact result:
Revised: Hence we have the exact result:

Original: [Equation]
Revised: [Equation]

Original: We know that the optimal target is C = − 1 2 η − 1 2 .
Revised: We know that the optimal target is C = − 1 2 η − 1 2 .

Original: Therefore we deﬁne the error δC =
Revised: Therefore we deﬁne the error δC =

Original: C + 12 η − 1 2 .
Revised: C + 12 η − 1 2 .

Original: Linearizing Equation (13) gives us:
Revised: Linearizing Equation (13) gives us:

Original: [Equation]
Revised: [Equation]

Original: We were not yet able to ﬁnd a general solution of this equation, but we can obtain a simple result for the trace y t .
Revised: We were not yet able to ﬁnd a general solution of this equation, but we can obtain a simple result for the trace y t .

Original: = tr( δC ) at time t :
Revised: = tr( δC ) at time t :

Original: [Equation]
Revised: [Equation]

Original: We therefore have a asymptotic linear convergence: y t ∝ e − 2 t y 0 which is independent of the parameters of the Gaussian model.
Revised: We therefore have a asymptotic linear convergence: y t ∝ e − 2 t y 0 which is independent of the parameters of the Gaussian model.

Original: We can also get an non-asymptotic estimate of a speciﬁc error measure for the precision matrix.
Revised: We can also get an non-asymptotic estimate of a speciﬁc error measure for the precision matrix.

Original: Using equation (13), we have the following dynamics for the precision C − 1 :
Revised: Using equation (13), we have the following dynamics for the precision C − 1 :

Original: [Equation]
Revised: [Equation]

Original: Taking the trace
Revised: Taking the trace

Original: [Equation]
Revised: [Equation]

Original: Hence we get the following exact result:
Revised: Hence we get the following exact result:

Original: [Equation]
Revised: [Equation]

Original: which is again independent of the parameters of the Gaussian model.
Revised: which is again independent of the parameters of the Gaussian model.

Original: Additionally this tells us that if the covariance C is non-singular at time t = 0, it will remain non-singular for all t (tr( C − 1 ) would be inﬁnite).
Revised: Additionally this tells us that if the covariance C is non-singular at time t = 0, it will remain non-singular for all t (tr( C − 1 ) would be inﬁnite).

Original: Hence if we start with N > d particles with a proper empirical covariance, they cannot collapse to make C singular
Revised: Hence if we start with N > d particles with a proper empirical covariance, they cannot collapse to make C singular

Original: Appendix G. Fixed-points for Gaussian model ( N ≤ D )
Revised: Appendix G. Fixed-points for Gaussian model ( N ≤ D )

Original: Applying equation (11) to our ﬁxed point equation we get
Revised: Applying equation (11) to our ﬁxed point equation we get

Original: [Equation]
Revised: [Equation]

Original: N
Revised: N

Original: Hence, the set of centered positions of the particles S
Revised: Hence, the set of centered positions of the particles S

Original: = { x i − m } Ni =1 , are all eigenvectors of the matrix E
Revised: = { x i − m } Ni =1 , are all eigenvectors of the matrix E

Original: ˆ q (cid:2) g
Revised: ˆ q (cid:2) g

Original: ( x )( x − m ) (cid:62) (cid:3) with eigenvalue 1.
Revised: ( x )( x − m ) (cid:62) (cid:3) with eigenvalue 1.

Original: S spans a N − 1 dimensional space (we have (cid:80) Ni
Revised: S spans a N − 1 dimensional space (we have (cid:80) Ni

Original: =1 ( x i − m ) = 0).
Revised: =1 ( x i − m ) = 0).

Original: If we specialise to a Gaussian target p ( x ) =
Revised: If we specialise to a Gaussian target p ( x ) =

Original: N ( x | µ, Σ), we have g ( x ) =
Revised: N ( x | µ, Σ), we have g ( x ) =

Original: − 2 η 2 x − ηand can reuse the result from Equation (12):
Revised: − 2 η 2 x − ηand can reuse the result from Equation (12):

Original: [Equation]
Revised: [Equation]

Original: which shows that the obtained low-rank covariance C and the target covariance Σ have
Revised: which shows that the obtained low-rank covariance C and the target covariance Σ have

Original: N − 1 eigenvectors and eigenvalues in common.
Revised: N − 1 eigenvectors and eigenvalues in common.

Original: Appendix H. Additional experiments H.1. Gaussian Process Classiﬁcation
Revised: Appendix H. Additional experiments H.1. Gaussian Process Classiﬁcation

Original: We turn to estimating the quality of the variational distribution given a varying number of particles for a non-Gaussian posterior.
Revised: We turn to estimating the quality of the variational distribution given a varying number of particles for a non-Gaussian posterior.

Original: Speciﬁcally, we consider a GP classiﬁer: y ∼
Revised: Speciﬁcally, we consider a GP classiﬁer: y ∼

Original: Bernoulli( σ ( f )) , where σ is the logistic function on the Ionosphere dataset (Sigillito et al., 1989) from the UCI repository (Dua and Graﬀ, 2017).
Revised: Bernoulli( σ ( f )) , where σ is the logistic function on the Ionosphere dataset (Sigillito et al., 1989) from the UCI repository (Dua and Graﬀ, 2017).

Original: This dataset contains 351 instances and 34 features and is split into a train/test set with 231/120 instances.
Revised: This dataset contains 351 instances and 34 features and is split into a train/test set with 231/120 instances.

Original: The dimensionality of a GP is equal to the number of training instances.
Revised: The dimensionality of a GP is equal to the number of training instances.

Original: However, our method does not decrease the cubic complexity, as the GP prior computations are still the bottleneck.
Revised: However, our method does not decrease the cubic complexity, as the GP prior computations are still the bottleneck.

Original: We sample from the posterior using the Gibbs sampling approach from Wenzel et al.
Revised: We sample from the posterior using the Gibbs sampling approach from Wenzel et al.

Original: and train a VI model using the method described in Hensman et al.
Revised: and train a VI model using the method described in Hensman et al.

Original: For all experiments the kernel parameters are ﬁxed during training.
Revised: For all experiments the kernel parameters are ﬁxed during training.

Original: We train GPF starting with a MAP estimation and we estimate the predictions on a test .
Revised: We train GPF starting with a MAP estimation and we estimate the predictions on a test .

Original: To compare the accuracy of the results, we compute the
Revised: To compare the accuracy of the results, we compute the

Original: Wasserstein distance with the L 2 norm, using the stochastic approach from Genevay et al.
Revised: Wasserstein distance with the L 2 norm, using the stochastic approach from Genevay et al.

Original: The Wasserstein metric has the advantage of allowing us to compare both particlebased method and parameter-based methods.
Revised: The Wasserstein metric has the advantage of allowing us to compare both particlebased method and parameter-based methods.

Original: Figure H.1 compares accuracy, negative loglikelihood and the Wasserstein metrics of GPF with Markov chain Monte Carlo (MCMC) (used as the ground truth).
Revised: Figure H.1 compares accuracy, negative loglikelihood and the Wasserstein metrics of GPF with Markov chain Monte Carlo (MCMC) (used as the ground truth).

Original: The results show that GPF is able to match the accuracy results of VI for a suﬃciently large number of particles.
Revised: The results show that GPF is able to match the accuracy results of VI for a suﬃciently large number of particles.

Original: As expected, with a number of particles that closely matches the problem dimensionality (i.e. 231 the training set size), GPF closely matches the performance of VI.
Revised: As expected, with a number of particles that closely matches the problem dimensionality (i.e. 231 the training set size), GPF closely matches the performance of VI.

Original: Moreover we see that an increasing number of particles progressively reduces the Wasserstein distance between our variational distribution and the ground truth.
Revised: Moreover we see that an increasing number of particles progressively reduces the Wasserstein distance between our variational distribution and the ground truth.

Original: H.2. Bayesian Neural Networks
Revised: H.2. Bayesian Neural Networks

Original: Finally, we evaluate our method on the challenging problem of inference with high-dimensional
Revised: Finally, we evaluate our method on the challenging problem of inference with high-dimensional

Original: Bayesian Neural Networks (BNN) .
Revised: Bayesian Neural Networks (BNN) .

Original: We consider the LeNet convolutional neural network (LeCun et al., 1998) trained on MNIST (LeCun and Cortes, 2010) implemented in
Revised: We consider the LeNet convolutional neural network (LeCun et al., 1998) trained on MNIST (LeCun and Cortes, 2010) implemented in

Original: Flux.jl Innes (2018).
Revised: Flux.jl Innes (2018).

Original: We ﬁrst train LeNet until we get the maximum likelihood ( ML ) estimate; from there, we ﬁx the weights of the convolutional layers and focus on estimating the weights of the dense layers (the weights of the convolutional layers are notoriously difﬁcult to infer; see Gal and Ghahramani (2015) and Shridhar et al.
Revised: We ﬁrst train LeNet until we get the maximum likelihood ( ML ) estimate; from there, we ﬁx the weights of the convolutional layers and focus on estimating the weights of the dense layers (the weights of the convolutional layers are notoriously difﬁcult to infer; see Gal and Ghahramani (2015) and Shridhar et al.

Original: The problem is thus very high-dimensional, with 41,854 parameters (i.e., the number of weights in the dense layers).
Revised: The problem is thus very high-dimensional, with 41,854 parameters (i.e., the number of weights in the dense layers).

Original: We compare GPF to SWAG (Maddox et al., 2019) and GVA.
Revised: We compare GPF to SWAG (Maddox et al., 2019) and GVA.

Original: For SWAG we do not place a prior on the weights, use the momentum optimizer with constant learning rate ( η = 0 . 05) and use the last 50 samples taken every 4th of an epoch.
Revised: For SWAG we do not place a prior on the weights, use the momentum optimizer with constant learning rate ( η = 0 . 05) and use the last 50 samples taken every 4th of an epoch.

Original: For GPF we use 50 particles and consider two MF variants: Structured MF by assuming independence between layers and MF by assuming independence between each weight.
Revised: For GPF we use 50 particles and consider two MF variants: Structured MF by assuming independence between layers and MF by assuming independence between each weight.

Original: For GVA we use
Revised: For GVA we use

Original: 50 samples for each expectation estimation and assume independence between weights, as even a structured approach would be prohibitively expensive.
Revised: 50 samples for each expectation estimation and assume independence between weights, as even a structured approach would be prohibitively expensive.

Original: We compare diﬀerent models in terms of standard performance metrics, namely negative log-likelihood and accuracy, as well as in terms of calibration.
Revised: We compare diﬀerent models in terms of standard performance metrics, namely negative log-likelihood and accuracy, as well as in terms of calibration.

Original: To compute the calibration we use the technique from Guo et al.
Revised: To compute the calibration we use the technique from Guo et al.

Original: (2017): we take the maximum probability among all classes and split them into 20 bins of equal frequency.
Revised: (2017): we take the maximum probability among all classes and split them into 20 bins of equal frequency.

Original: We then compute the accuracy of the samples contained in each bin.
Revised: We then compute the accuracy of the samples contained in each bin.

Original: In a perfectly calibrated model, the obtained probability and the accuracy should be the same.
Revised: In a perfectly calibrated model, the obtained probability and the accuracy should be the same.

Original: Figure H.1 compares the competing approaches in terms of the calibration of the predictions, showing that GPF achieves considerably better performance than GVA, and perform similarly to ML and SWAG.
Revised: Figure H.1 compares the competing approaches in terms of the calibration of the predictions, showing that GPF achieves considerably better performance than GVA, and perform similarly to ML and SWAG.

Original: Figure H.1 compares the averaged negative log-likelihood and the accuracy when varying the value of the prior standard deviation α .
Revised: Figure H.1 compares the averaged negative log-likelihood and the accuracy when varying the value of the prior standard deviation α .

Original: We can also observe that having a structured MF approach (even low-rank) improves both on the calibration and on the quality of the obtained posterior.
Revised: We can also observe that having a structured MF approach (even low-rank) improves both on the calibration and on the quality of the obtained posterior.

Original: SWAG generally outperforms competitors but is also less ﬂexible and less principled.
Revised: SWAG generally outperforms competitors but is also less ﬂexible and less principled.

Original: Unlike
Revised: Unlike

Original: GPF, it can only be used in the context of SGD and cannot be applied to tackle general
Revised: GPF, it can only be used in the context of SGD and cannot be applied to tackle general

Original: Bayesian inference problems.
Revised: Bayesian inference problems.

Original: Interestingly, despite being a general purpose method GPF is able to achieve on par calibration with specialized techniques.
Revised: Interestingly, despite being a general purpose method GPF is able to achieve on par calibration with specialized techniques.