Paper:
Decoupled Variational Gaussian Inference
Variational Gaussian (VG) inference methods that optimize a lower bound to the marginal likelihood are a popular approach for Bayesian inference. A difficulty remains in computation of the lower bound when the latent dimensionality L is large. Even though the lower bound is concave for many models, its computation requires optimization over O(L 2 ) variational parameters. Efficient reparameterization schemes can reduce the number of parameters, but give inaccurate solutions or destroy concavity leading to slow convergence. We propose decoupled variational inference that brings the best of both worlds together. First, it maximizes a Lagrangian of the lower bound reducing the number of parameters to O(N ), where N is the number of data examples. The reparameterization obtained is unique and recovers maxima of the lower-bound even when it is not concave. Second, our method maximizes the lower bound using a sequence of convex problems, each of which is parallellizable over data examples. Each gradient computation reduces to prediction in a pseudo linear regression model, thereby avoiding all direct computations of the covariance and only requiring its linear projections. Theoretically, our method converges at the same rate as existing methods in the case of concave lower bounds, while remaining convergent at a reasonable rate for the non-concave case.

Review:
The paper presents new techniques, and theoretical results relating to those techniques, to optimise the Variational Gaussian lower bound to the log evidence in latent Gaussian models LGMs. The technique could be applied to a broader category of latent linear models but their presentation focuses on LGMs only.VG approximate inference is important because it has many nice properties : it is widely applicable, often quite accurate and relatively fast.This paper attempts to makes VG methods more scalable without resorting to making factorisation assumptions on the approximating Gaussian distribution.Whilst the authors show how the objective function can be `decoupled' they do not show experimentally that this leads to a clear improvement in speed or scalability over standard techniques. Their largest problem has only 1.5k data points. I can only imagine that this because there is some scalability issue in their approach/implementation.Since the covariance of the VG approximation is not explicitly maintained i think it is critical that the authors explain to readers how expectations with respect to this density can be calculated once their solver has found the optimum. We are not just interested in the value of the bound but the moments of the approximating Gaussian also.The derivations of the theoretical results should be presented in full in the supplementary material. The theoretical results are quite difficult to follow. Whilst their theorectical contributions are interesting I wonder how useful they are -- solving a lagrangian with non-linear constraints is difficult. Their linearisation method is not guaranteed to converge.Furthermore, one of the nice things about Seeger's local variational bounding method is that you could specify the rank of the covariance approximation. Could you do that in this context? Or is the entire covariance always approximated -- in which case how scalable is this method anyway. QualityThe experiments could be more convincing. This paper is about making VG inference scalable and fast? The largest problem to which their method is applied has only 1500 data points -- I wouldn't classify this as large. At this scale they only achieve a marginal improvement over the Cholesky method. The experiments do not detail the stopping conditions used on either algorithm and so making comparisons is difficult. Also, it is difficult to see what numeric bound values were achieved for either method. I think these results would be better presented in a table, with means and standard errors over multiple runs.It is especially important that their experiments show the results obtained over multiple runs since their solver is not guaranteed to converge.ClarityThe authors need to spend a bit more time and effort making the presentation cleaner.I think the derivations should be presented in full in the supplementary material taking time to describe the notation used and steps taken. As it stands the descriptions of the derivations in the main body of the paper are too short to understand.OriginalityThis paper presents novel results about nature of the VG bound optimisation problem. However, the usefulness of these results is unclear -- how fast would the solvers that are guaranteed to converge take in practice? If you only use a solver that isnt guaranteed to converge what is the importance of these results.SignificanceThis paper presents some novel results about the VG approximate inference optimisation problem. Minor points and typos.line 45 : Maybe Gaussian processes are not the best example of models with strong correlations between variables in this context since we would only attempt approximate inference for GP models when O(D^2) space and O(D^3) time is feasible and so wouldnt need a distributed solver.* Section 2 para 1. I appreciate that it is convenient to focus on just LGMs in this paper but i think readers would appreciate a more precise description of the broader class of models that your method could be applied to.Eq. (5) maybe explain the log det V notation.line 104: expand with no sline 134: it doesnt make sense to introduce auxiliary variables such that h_n = \bar{m_n}. What are the constraints for then? Either you introduce the auxiliary variables and the constraints or you just set h_n to be defined as \bar{m}_n. line 141: concave and convex mix up? should it not say ...function f (concave with respect to Cholesky of V).eq(6) Maybe say subject to constraints ... for n=1,..,N.Theorem 4.1 eq 8 -- you should explain that \Lambda^* = \diag(\lambda^*) .line 159. missing n subscript on \lambda^*eq (9) : i think notation used for the partial derivatives is a little unclear. You either need to explain the notation used or write the partial derivate out explicitly with a vertical "evaluated at" bar with subscript h=h^*  line 169 : the condintions that paragraph 1 page 4 : maybe change each of the "The x part..." to "Part x ..."line 177 : constrain ", line 177 : this sentence is weird ", and similar to [9]"line 179: "while the second equality is simply change of variables" line 190: there are two prerequisites...lines 194 to 202 : This is hard to follow. I would recommend taking your time, going through each of the steps explicitly and precisely defining terms in the supplementary material.line 211 : this sentence is poor.line 219 : "this optimisation appears..." which optimisation? Both formulations are difficult?line 229 : "and is an effective for " method?line 254 : "for primal function" for the primal function?line 260 : missing super script k on f ? line 266 : of the dual variables are equalline 285 : two eithers.line 291 : see Algorithm 1?line 293 : quadratic convergence under what conditions. The next sentence is missing a "this" it is also unclear.line 294: we never observe this in line 312 : Sentence starting "Basically, ..." What is the justification for this heuristic?Bibliography : Authors in references need to be made consistent