Paper: Hybrid Kernel Stein Variational Gradient Descent (ICLR-2024)

Reviewer comment (Novelty):
The analysis seems like straightforward extensions of existing results, e.g., most proofs in Section 4 are 1-liners.

Author response:
We acknowledge that some proofs are presented as straightforward extensions and one-liners. However, we would like to mention some key technical details in the Appendix. Lemma B.1 is non-trivial and is required to establish the optimality of the h-SVGD update direction. Some oversights in proofs in the existing literature have been addressed. For example, our Assumption (B2) replaces a previous very restrictive assumption which, through Lemma B.2, makes Proposition 4.7 a stronger result than its counterpart in Korba et al. (2020). Also, Remark 3 addresses the interchangeability of expectation and inner products, which has not been addressed in the SVGD literature to our knowledge.

Reviewer comment (Experiments):
While the result (Thm 4.1) on the optimality of the h-SVGD update direction and many theoretical results does not require $k_1$ to be related to $k_2$, in all experiments, $k_2$ is simply a scalar multiple of $k_1$, i.e., $k_2 = f(d)k_1$. This seems to suggest the more general theory does not give rise to diverse choices of $k_1$ and $k_2$ in applications.

Author response:
Diversity of kernel choice: As noted, the developed theory does not require kernels $k_1$ and $k_2$ to be related, but the simulation results were based on kernels of the form $k_2=f(d)k_1$. We have now included additional BNN experiments with a range different kernels in Appendix D. Please see Main Response for further details.

Reviewer comment (Experiments):
Even if we only consider the case of $k_2 = f(d)k_1$, it remains a question of how to choose the scaling function $f(d)$. The paper suggests taking $f(d) = \sqrt{d}$ or $\ln(d)$, but further comparision (either empirical or theoretical) is lacking.

Author response:
The current work serves to establish the theory of h-SVGD for the first time, and to demonstrate its ability to alleviate variance collapse. However, we agree that choosing the scaling function is a worthwhile direction for future research, and we now explicitly mention this in Section 5.2 and the conclusion.

Reviewer comment (Theory):
As discussed in Appendix C, h-KSD is not a valid discrepancy measure, which seems to suggest it is less useful as a metric than the vanilla KSD.

Author response:
While, as correctly noted, h-KSD is not a valid discrepancy measure, the descent lemma (Theorem 4.2) shows that the KL divergence bounds can be written in terms of the vanilla KSD and guarantees a decrease in the KL divergence at every step. Some additional discussion has been added to Section 4.3. See the Main Response for further discussion.

Reviewer comment (Theory):
Could the authors explain why $\phi_{\mu, p}^{k_1,k_2} \in \mathcal{H}_1^d\cap \mathcal{H}_2^d$ in Thm 4.1?

Author response:
The set within which the optimisation takes place is chosen as a subset of this intersection. For standard kernel choices, Remark 2 ensures that this intersection will be either $\mathcal H_1$ or $\mathcal H_2$. This amounts to optimising within a unit ball on $\mathcal H_1$ or $\mathcal H_2$, although with a norm induced by the sum Hilbert space.

Reviewer comment (Theory):
Are there any applications of choices of $k_1$ and $k_2$ such that $\mathcal{H}_1$ and $\mathcal{H}_2$ does not include one another?

Author response:
To our knowledge, the SVGD literature does not contain kernel choices of this type.

Reviewer comment (Experiments):
How is the bandwidth of the kernels affecting the variance collapse, compared to the choice of $f(d)$? Or to put the question in another way, how to simultaneously choose the bandwidth and $f(d)$ in applications?

Author response:
If $k_2$ is the RBF kernel, a larger bandwidth allows a steeper gradient $\nabla k_2$ when evaluated on distant particles. For intuition, think of the plot of $\exp(-x^2/h)$, so that a higher bandwidth increases the repulsive force, and thereby mitigates variance collapse. We now provide a comment on this in Appendix D.

Reviewer comment (Evaluation):
In Table 1, the test accuracy on Yacht for h-SVGD is poor compared to SVGD. Why is this the case?

Author response:
One possible reason is that the Yacht dataset has the smallest number of records (306) and the second smallest number of features (6) of the ten datasets considered.

Reviewer comment (Writing):
Where is the map $\Phi_p^{k_1,k_2}$ defined in (9)?

Author response:
The paper has been updated with the definition of $\Phi_p^{k_1,k_2}$ now just before equation (9).

Reviewer comment (Novelty):
The paper focuses on h-SVGD, which is fine, but I am not convinced about the impact of this SVGD variant. The empirical results in this paper do not show a conclusive advantage for preferring this method over the standard SVGD, and the same applies to the original paper by D’Angelo et al., (2021).

Author response:
Please see our Main Response for full details. We have now improved our presentation of our empirical results in Section 5 to more clearly demonstrate that h-SVGD has similar performance to SVGD in terms of test RMSE and test log-likelihood, but improved (or equivalent) variance estimation. This is now most clearly seen in Figure 1, revised Figure 2, and new Figure 5 in Appendix D, which summarises performance over 9 kernel variants.

Reviewer comment (Theory):
Following the last point, although the scope of the paper is to provide a theoretical ground for h-SVGD, perhaps it will have a stronger contribution if the authors would clearly state (and evaluate) families of valid kernels for the repulsive term.

Author response:
Thank you for this welcome suggestion. We have now added results using additional different kernel families (9 variants in total) for the repulsive kernel to Appendix D, including the kernel adopted in D'Angelo et al. (2021). The theory still applies to this latter kernel choice; in particular, the conditions for the descent lemma (Theorem 4.2) are satisfied, as detailed in Appendix D.

Reviewer comment (Evaluation):
I find it odd that the test log-likelihood is not correlative with the dimension averaged marginal variance. If indeed the particles are more diverse with h-SVGD then I expected that it will be reflected in a better test log-likelihood.

Author response:
To our understanding, log-likelihood is a measure of test accuracy. Better variance representation doesn't necessarily mean that log-likelihood is improved.

Reviewer comment (Writing):
The method section is not written clearly enough in my opinion. Specifically, the authors can provide better intuition for some of the results. Also, perhaps the authors should present only the main claims in the main text and provide a proof sketch for them.

Author response:
We have now included some additional commentary in Section 4 to provide more intuition on the results. In particular, we note Remark 1 and Remark 2 (which has now been moved slightly earlier in the paper) for some intuition on the main result (Theorem 4.1). Further discussion has also been added around Theorem 4.2 on how the KL divergence will always decrease under h-SVGD.

Reviewer comment (Theory):
In D’Angelo et al., (2021) the authors used a different kernel for the repulsive term from the ones used in this paper. Is there something in the theory that does not apply on their kernel? It may be interesting to evaluate the performance and variance shrinkage of that kernel as well.

Author response:
As requested we have now included simulations using this kernel for the repulsive term, as outlined above.

Reviewer comment (Novelty):
My primary concern pertains to the apparent significance of the hybrid kernel approach, as presented in the paper. The paper suggests that the hybrid kernel is proposed as a solution to circumvent the issue of variance collapsing. Nonetheless, it should be noted that there are numerous preceding studies such as S-SVGD, Grassman SVGD, among others, addressing similar challenges. Some of these methods have successfully established a proper goodness-of-fit test, ensuring that the resultant discrepancy is a valid one.

Author response:
Significance of h-SVGD: Please see our Main Response for full details. We have now improved our presentation of our empirical results in Section 5 to more clearly demonstrate that h-SVGD has similar performance to SVGD in terms of test RMSE and test log-likelihood, but improved (or equivalent) variance estimation. This is now most clearly seen in Figure 1, revised Figure 2, and new Figure 5 in Appendix D, which summarises performance over 9 kernel variants. In addition, at the start of Section 5 we now comment that other previous methods (such as GSVGD and S-SVGD) also tackle variance collapse, but they do so at a far greater computational cost. In particular, S-SVGD requires additional computation of the optimal test directions and GSVGD requires additional computation at each step to update the projectors. The advantage of h-SVGD is that there is no added computational cost over regular SVGD.

Reviewer comment (Theory):
My second concern revolves around the convergence properties of the h-SVGD algorithm. The manuscript demonstrates that the descent magnitude is h-KSD, which, as acknowledged, is not a proper discrepancy. This raises questions regarding the algorithm’s capability to minimize the KL divergence effectively, specifically, whether it can drive the KL divergence to zero. A descent magnitude (h-KSD) of zero does not imply that the distributions are equal or that the KL divergence has been minimized to zero.

Author response:
Convergence properties of h-SVGD:
The question is about whether h-SVGD can minimise the KL divergence effectively: see our Main Response for full details. As noted, the descent magnitude is h-KSD (Theorem 4.1), which is not a proper discrepancy. However, the descent lemma (Theorem 4.2) bounds the decrease in KL divergence in terms of a proper discrepancy, namely the KSD of one of the kernels. We have now included further details in Section 4.3 that show for proper choice of step size, the KL divergence is strictly decreasing at all times, meaning that the h-SVGD algorithm avoids cases where $\mathbb S_{k_1,k_2}(\mu_\ell^\infty,\nu_p)=0$ but $\mu_\ell^\infty$ and $\nu_p$ are not equal almost everywhere.

Reviewer comment (Novelty):
There is a need to understand how h-SVGD, with its unique convergence characteristics, stands out amidst other existing methodologies addressing similar issues, especially in light of the previous point on the hybrid kernel approach’s advantages.

Author response:
Significance of h-SVGD: Please see our Main Response for full details. We have now improved our presentation of our empirical results in Section 5 to more clearly demonstrate that h-SVGD has similar performance to SVGD in terms of test RMSE and test log-likelihood, but improved (or equivalent) variance estimation. This is now most clearly seen in Figure 1, revised Figure 2, and new Figure 5 in Appendix D, which summarises performance over 9 kernel variants. In addition, at the start of Section 5 we now comment that other previous methods (such as GSVGD and S-SVGD) also tackle variance collapse, but they do so at a far greater computational cost. In particular, S-SVGD requires additional computation of the optimal test directions and GSVGD requires additional computation at each step to update the projectors. The advantage of h-SVGD is that there is no added computational cost over regular SVGD.

Reviewer comment (Theory):
For theorem 4.1, how do you ensure the $H_1 \cap H_2$ is not empty?

Author response:
Remark 2 mentions that either $\mathcal H_1 \subseteq \mathcal H_2$ or $\mathcal H_2 \subseteq \mathcal H_1$ for many common choices of kernel (e.g. RBF, IMQ, log-inverse, or Matérn). This includes when $k_1$ and $k_2$ are from different families mentioned above. So the intersection $\mathcal H_1 \cap \mathcal H_2$ will be either $\mathcal H_1$ or $\mathcal H_2$. We are not aware of applications in the SVGD literature that use kernels other than those listed above. Remark 2 has been reworded slightly and moved so that it appears just after Theorem 4.1.

Reviewer comment (Experiments):
Despite rigorous theoretical results, the experimental results are not sufficient to show that it mitigates the variance collapse issue better than previous methods (e.g. S-SVGD and G-SVGD).

Author response:
Strong experimental results for h-SVGD: Please see our Main Response for full details. However, note that the mitigation of variance collapse in high dimensions is demonstrated in Section 5.1 and Figure 1 (a-c), which shows a significant improvement in estimating the true variance in high dimensions over SVGD, even when the number of dimensions is greater than the number of particles ($\gamma>1$). Also, revised Figure 2 now more clearly demonstrates an increase in DAMV for the BNN example, with consistent performance in test RMS and log-likelihood. Please also see new Figure 5 in Appendix D for additional qualitatively similar results with other kernel choices (9 kernel variations). Overall, the general performance of h-SVGD is comparable with SVGD but with the advantage that h-SVGD has superior performance in mitigating variance collapse.

Reviewer comment (Experiments):
It would be useful to study the variance collapse issue with inference tasks in higher dimensions in comparison to previous approaches, such as the experiments in [1], as this is mainly an issue that arises in large dimensions.

Author response:
While the primary aim of this paper is to provide theoretical justification of h-SVGD, we recognise that it is useful to support this by also demonstrating the performance of h-SVGD in areas that are directly related to this theoretical development; namely performance with regard to different families of kernels. This has now been done, and has been implemented in Appendix D. We feel that comparison to other competitor methods (such as GSVGD and S-SVGD) is somewhat adjacent to the focus of this paper, and is more relevant for research that makes algorithmic contributions in this field.

Reviewer comment (Evaluation):
What is the computational cost of h-SVGD compared to SVGD? Is it the same or more expensive?

Author response:
In terms of computational costs: the cost of updating the particles for both SVGD and h-SVGD is $O(N^2)$ at each step. We have added this statement at the beginning of Section 5. The SVGD update can be slightly faster when $k(x,y)$ can be reused to compute $\nabla k(x,y)$ (this applies to the RBF kernel for example), but the order is still $O(N^2)$.