Paper: RASP Quadratures: Efficient Numerical Integration for High-Dimensional Mean-Field Variational Inference (ICLR-2024)

Reviewer comment (Novelty):
The related work section should be expanded to include more discussion. Clearly, QMC is very relevant and other quadrature methods are also relevant.

Author response:
- Agreed. We substantially revised the related work section to provide a more thorough overview these methods to help place our work in a clearer context.

Reviewer comment (Theory):
Is there any theoretical justification for the number of nodes (and corresponding function evaluations) in general?

Author response:
- Theorem 2 shows how the number of nodes affects the number of bivariate quadratic basis functions that integrate exactly. That said, the optimal number of nodes really depends on time / computational constraints and the characteristics of the loss topography.

Reviewer comment (Novelty):
The paper lacks reference to a number of highly relevant related work that develops a context for the current paper. Specifically, in Section 1 first paragraph, a citation is needed for Markov-chain Monte Carlo (MCMC). Although this is folk wisdom, a citation is required for the claim "MCMC is not practical for large learning architectures that often contain billions of parameters."

Author response:
- We added several references to support this claim.

Reviewer comment (Writing):
In Section 1 second paragraph, a citation is needed for mean-field variational inference (MFVI). MFVI did not appear out of thin air, and the development/popularization of MFVI is often attributed to references [1-3].

Author response:
- We were aware of these references and should have added them earlier.

Reviewer comment (Writing):
In Section 2.1 first sentence, a citation is needed for variational inference. In general, two well-known reviews [7,8] are cited for this.

Author response:
- Thank you. We added these well-known sources and a recent survey by Zhang (2018).

Reviewer comment (Novelty):
Probably due to the missing citations above, the paper misses a very important line of related works, including:
- Randomized quasi-Monte Carlo (RMQC) for variational inference [9]
- RQMC and stochastic quasi-Newton optimization for variational inference [10]  
The paper does not adequately put the work in the context of this line of work. It is unclear whether this method improves over these works or is different from the (randomized) quasi-Monte Carlo schemes used in them.

Author response:
- We appreciate these references and we also performed additional searching to discuss these contributions in more detail under related work.

Reviewer comment (Experiments):
The choice of baselines and the overall design of the experiments make the conclusion of the experiments unclear. The experiments do not appear to be doing an apple-to-apple comparison, and some experimental details are missing.

Author response:
- RASP quadratures target improved integration of basis functions that dominate QNVB updates. This is why our original experiments examined several architectures and other variance-reduced monte carlo methods with QNVB. That said, a quadrature comparison with other training methods is helpful and we added a battery of tests using SGVB, as suggested. We had to move comparisons of non-VI training methods into the appendix, but that isn't problematic since it isn’t the central focus of the paper. We also added additional experiment details to the appendix.

Reviewer comment (Reproducibility):
Regarding experimental details, since variational inference is fundamentally a method for approximating the posterior, it is unclear how the joint likelihood was obtained here. Was a prior set on the parameters of these models? Was an improper flat prior set on the parameters? What were the hyperparameters? The choice of prior in deep learning models is known to affect results significantly [11].

Author response:
- We used uniform priors to avoid an additional complication, but regularization techniques can easily be incorporated into our pipeline. As you are almost certainly aware, L1 and L2 terms can be regarded as negative log priors. Other hyperparameters are described in the appendix. We added a paragraph about this under the experiments section.

Reviewer comment (Experiments):
The paper only compares against the following combinations: RASP + QNVB, MC + SGVB, MC + QNVB, QMC + QNVB. It does not compare RASP + SGVB. Also, closely related works based on randomized QMC [9,10] and the QNVB algorithm proposed in [10] should have been included.

Author response:
- RQMC methods typically use more function evaluations than our target range (10+ as opposed to 3 to 6) and they are not designed to target second-order exactness. Such comparisons would be interesting, but they also open a different line of investigation from the role of exactness in QNVB updates. Please see our updated related work.

Reviewer comment (Theory):
When denoting the variational approximation q, it is non-standard to denote conditioning on $\varphi$, because the variational parameters $\varphi$ are not considered to be random variables (their probability density has not been defined).

Author response:
- Our notation follows Zhang (2018) to explicitly clarify that the shape of the variational density is determined by phi, which is the actual variable of optimization. This notation is helpful for projective integral updates, since phi comprises the projection coefficients that RASP quadratures target. See equations 3 and 4.

Reviewer comment (Writing):
In Section 1.1 third paragraph second from the last sentence, the text says: “… Hessian diagonal of the training loss, which can be expensive to compute.” It would be more accurate to state that it is noisy. Noisy estimates are cheap to obtain, as shown by [12].

Author response:
- This is a fair point and we have made the modification.

Reviewer comment (Theory):
In Section 2.2 last sentence, the scalability claim is unclear. Is there any evidence that alternative integration schemes scale better in terms of dimensionality? Quasi-Monte Carlo methods, while converging faster with respect to the number of samples, have an explicit dimension dependence, unlike regular Monte Carlo. RASP likely has similar behavior, though no convergence bound akin to those in the QMC literature is presented. When the number of samples is small, it is uncertain whether the method can be claimed to be scalable in terms of dimension.

Author response:
- RASP quadratures are not unbiased estimators and they do not converge to the exact integral with increasing points. While they are exact on all linear basis functions, as well as a large fraction of second-order basis functions (theorems 1 and 2), there are systematic errors that persist for some higher-order basis functions. The point of interest regarding increased reference dimensionality is the fraction of second-order basis functions that become exact, which we have analyzed with theorems and proofs.

Reviewer comment (Novelty):
I am rather skeptical about novelty of the proposed quadrature. Having never worked in a setting in which even n = d + 1 nodes is prohibitive, I cannot provide any references. However, the limited and superficial literature review on pages 4 and 9 does not convince me that the authors know the relevant quadrature well enough to say with any confidence that something like this has not been tried before.

Author response:
“The limited and superficial literature review on pages 4 and 9 does not convince me that the authors know the relevant quadrature well enough to say with any confidence that something like this has not been tried before.”
 - This is a fair point, and we inappropriately disregarded the importance of citing relevant related work. We have diligently reviewed surveys, books, and new papers regarding high-dimensional integration, sigma-point methods, QMC methods, and RQMC methods. We have also substantially revised and lengthened our related work section accordingly to coherently place our contributions in context. While we still cannot be sure if this method (or something similar) has been published before, it is certainly not commonly discussed. We have added a statement to this effect.

Reviewer comment (Writing):
p3: The definition of \mathcal{F} in Eq (3) is unclear.

Author response:
“The definition of \mathcal{F} in Eq (3) is unclear.”
 - The basis depends on the exponential family that captures feasible variational densities. For the purpose of QNVB, the basis spans the set of univariate quadratics in R^d, which Eq (5) is intended to capture.

Reviewer comment (Writing):
p3: There are several squares of bolded quantities, some of which appear to be vectors [e.g., in Eq (5)]. What does squaring mean here?

Author response:
“What does squaring [of vectors] mean here?”
 - Powers on vectors are elementwise in these formulas and we have added a note to clarify this notation.

Reviewer comment (Writing):
p3: In the first paragraph of Sec 2.3 f: R^d \mapsto \R should use \to rather than \mapsto.

Author response:
“In the first paragraph of Sec 2.3 f: R^d \mapsto \R should use \to rather than \mapsto.”
 - Thank you!

Reviewer comment (Writing):
p4: "For quadrature formulas that obtain second-order exactness (integrating all quadratic total-degree polynomials exactly) over multivariate Gaussians, the evaluation nodes are called sigma points." It would be good to note that sigma point terminology is far from universal and only really used in non-linear filtering literature.

Author response:
"It would be good to note that sigma point terminology is far from universal and only really used in non-linear filtering literature.”
 - Good point. We have added this clarification.

Reviewer comment (Novelty):
p4: Given McNamee & Stenger (1967) and other old fully symmetric quadrature formulae, I doubt that Uhlmann (1995) was the first to design a second-order accurate quadrature rule.

Author response:
“Given McNamee & Stenger (1967) and other old fully symmetric quadrature formulae, I doubt that Uhlmann (1995) was the first to design a second-order accurate quadrature rule.”
 - Another good point; we have added this citation and removed the “original” qualifier.