Paper: Variational Learning of  Gaussian Process Latent Variable Models  through  Stochastic Gradient Annealed Importance Sampling (ICLR-2024)

Reviewer comment (Writing):
The notation $\tilde p(X, H)$ is a bit confusing since this is actually an estimator of the marginal $p(X)$, but it looks like a joint distribution.

Author response:
R:Thank you for your comment. We appreciate your careful review of our work. You are correct in pointing out this. After rechecking the symbol, we have corrected the notation $\hat{p}(H,K)$ to avoid confusion.

Reviewer comment (Writing):
Typo: undajusted.

Author response:
R: We have already corrected these typos. Please refer to the rebuttal revision for the updated version.

Reviewer comment (Writing):
No definition of the abbreviation: LV-GP or LVGP, MH, HMC.

Author response:
R: We have already corrected these typos. Please refer to the rebuttal revision for the updated version.

Reviewer comment (Experiments):
For Fig. 2 and Fig. 3, what about the reconstruction results from the other two baseline methods? And what about the latent space recovered by the other two baseline methods?

Author response:
Thank you for the reviewer's inquiry regarding Fig. 2 and Fig. 3. As per your request, we have included the reconstruction results from the other two baseline methods and the recovered latent space visualizations in the rebuttal revision. Please refer to the appendix section in our response for more detailed information

Reviewer comment (Presentation):
For Fig. 3 left, the bottom is the true data and the top row is the predicted? Figures in this paper are not explained clearly in their corresponding captions. Color usage, image order, method order, and etc sometimes make me confused.

Author response:
Thank you for the reviewer's question regarding Fig. 3. We apologize for the lack of clarity in the captions, which caused confusion. Indeed, in Fig. 3 left, the bottom row represents the true data, and the top row depicts the predicted data by our AIS method on the MINIst dataset. To address this issue, we will provide clearer explanations for all figures in the final version, ensuring that readers can accurately understand the color usage, image order, and method arrangement

We would like to inform the reviewer that we have made the necessary modifications as per your request. We have addressed concerns related to the clarity of figure captions and have taken steps to improve them for better understanding. Additionally, we have incorporated the requested visualizations for the reconstruction results from other baseline methods and the recovered latent space. We kindly request the reviewer to reconsider our paper in light of these revisions. Thank you for your time and valuable feedback.

Reviewer comment (Novelty):
The paper does not convincingly motivate potential shortcomings of existing methods that should be overcome by the described approach.

Author response:
We highly appreciate your concern about the motivation behind our proposed method and understand your skepticism. In this response, we will provide a detailed explanation of the motivation behind our proposed method and how it effectively overcomes the limitations of existing approaches. For more detailed information, please refer to the introduction section of this paper.

Our research motivation stems from the potential applications of GPLVMs in function estimation and unsupervised learning. Previous studies have mainly focused on training GPLVMs using variational inference (VI) methods and replacing the true probability terms with approximate posterior probability distributions to reduce model complexity.

However, we have found that in VI methods, there are still challenges in dealing with high-dimensional spaces and the increase in relative variance as the latent variable dimension increases, due to the limitations of standard importance sampling. These limitations make it difficult to construct a well-performing proposal distribution in high-dimensional spaces, thereby limiting the performance of existing methods.

To overcome these issues, we propose a new method called SG-AIS, based on the well-established theory of AIS [1]. The advantages of this method are primarily manifested in the following aspects:
- By introducing SG-AIS, we effectively address the challenges in high-dimensional spaces and reduce the increase in relative variance.
- The concept and design of SG-AIS are based on the AIS method, but with the introduction of stochastic gradient techniques and annealing strategies, we achieve higher efficiency and better convergence performance. Additionally, we approximate the posterior distribution using Langevin stochastic flow, which leads to significant improvements.
- We also design an efficient algorithm to reparameterize all variables in the evidence lower bound (ELBO) and propose a stochastic variant algorithm that uses data subsets to estimate gradients. This improves the speed and scalability of the algorithm.

We demonstrated the superiority of our approach through experiments on multiple benchmark datasets of GPLVMs. Our method not only exhibited tighter variational bounds and higher log-likelihood, but also demonstrated more robust convergence. These experimental results validate the effectiveness and feasibility of our proposed method.

Reviewer comment (Writing):
Algorithm 1: "stepsizes" rather than "stepsides".

Author response:
R: Thank you for identifying the typos in the paper. We appreciate your keen eye for detail, and we will make the necessary corrections. We apologize for any confusion caused by these errors

Reviewer comment (Evaluation):
How do you come to the conclusion that your method shows "more robust convergence" as claimed in the contributions at the end of the introduction section? Which experiment backs this claim? Figure 4 shows a strong peak shortly before 600 iterations and a strange decay at 350.

Author response:
The claim of "more robust convergence" in our contributions is based on the comparison of our proposed method with the baseline methods in terms of convergence behavior. The experiment that supports this claim is presented in Figure 4.

In Figure 4, we show the convergence curves for our method and the baseline methods over iterations. The strong peak shortly before 600 iterations that you mentioned is an expected behavior in the optimization process. It indicates that the model is actively exploring the solution space to find the optimal solution. The strange decay at 350 iterations may be due to fluctuations in the optimization process, but it does not affect the overall convergence behavior and the final performance of our method. As we described in the main text：''This can be attributed to the fact
that, by adding Langevin transitions, the algorithm’s variational distribution gradually moves from
the current distribution towards the true posterior distribution, resulting in sudden drops in the loss
function when reaching the target distribution. Thus, such phenomena can be regarded as a common
feature of annealed importance sampling and it becomes even more obvious in high-dimensional
datasets.''

Reviewer comment (Experiments):
Is it fair to compare MSE and NELL after a fixed number of iterations? One should either fix the computational budget or compare after convergence. What happens if you evaluate at 5000 iterations?

Author response:
As mentioned before, we have already analyzed the time complexity of AIS compared to IW on the Frey Faces dataset. Our proposed method does not show an increase in time complexity compared to the baseline method IW (and sometimes even lower). Therefore, even though we used a fixed number of iterations, we can ensure the fairness of the experiments. Regarding your mention of the results after 5000 iterations, I have already obtained preliminary results, and our method still outperforms the previous method. In response to your feedback, we will include these results, even the results after convergence, in the revised article to address your concerns.

Reviewer comment (Reproducibility):
The usage of annealing techniques as well as MC samplers can mean strong computational demands, especially in high dimensional settings and in complex models, particularly when dealing with high-dimensional data or complex models. Several experiments are conducted in related settings, and scalability results are only reported in terms of iterations while no information on the hardware used is provided. I expect the authors to provide this information in the final version of the draft.

Author response:
We have updated our experiments and conducted them on a Tesla A100 GPU in the revised "rebuttal revision".

Reviewer comment (Reproducibility):
On the same line as the previous point, I expect the authors to release some version of the code used to produce these results. This is not mentioned in the text and I deem reproducibility to be considered an important factor.

Author response:
In fact, we plan to publicly release the code and experimental setup related to our research paper upon publication. We will ensure that the code is user-friendly and comprehensible so that other researchers can validate and reproduce our experimental results.

Reviewer comment (Experiments):
The selection of hyperparameter values appears to play a crucial role in the performance of the presented method. Finding optimal values for parameters such as step size ($\eta$) or the selected number of bridging densities ($K$) may prove to be expensive and quite important in the final results. Moreover, even though it appears to be more restricted due to their usage, the choice for the evolution of $\beta_k$ coefficients, while contained in $\[0,1\]$ needs to be properly crafted as well.

Author response:
In Supplementary Material Appendix D of our paper, we have provided the settings for the number of bridging densities K, step size ($\eta$), and coefficients ($\beta_k$). However, we acknowledge that finding optimal values for these parameters still poses challenges, and we look forward to future work addressing your concerns.

Reviewer comment (Writing):
There seem to be some small typos throughout the text such as "fisrt" on page 4 or using "d" instead of "D" for dimensions on page 7 (e.g. "2d projections" instead of "2D"). I do not consider these important corrections at all since the text is very clean, I only suggest doing a final pass to fix these tiny mishaps.

Author response:
We have corrected these typos and will perform a final check.

Reviewer comment (Experiments):
How sensitive is the method to different choices of reparameterizations?

Author response:
R2： While we have not conducted additional experiments specifically addressing this sensitivity issue, we have carefully considered and compared several reparameterization choices in our original study. We believe that the findings presented in our paper can provide insights into the impact of these choices on the method's performance. As you previously mentioned, the reparameterization method, such as Euler-Maruyama discretization, ''leads to a simpler gradient computation and therefore an easier process to optimize model parameters during training'' . However, we acknowledge that further investigations and experiments can certainly enhance our understanding of the influence of different reparameterizations. We will take this suggestion into account for future studies.

Reviewer comment (Theory):
Can you provide more insights into how the proposed annealing process affects the exploration of posterior distributions? What are some potential trade-offs or considerations when choosing an appropriate annealing schedule?

Author response:
The proposed annealing process in SG-AIS plays a crucial role in exploring the posterior distributions.

In the context of this paper, the posterior distribution refers to the distribution of the latent variables given the observed data. This distribution is often intractable and challenging to sample from directly. SG-AIS aims to approximate this posterior distribution by transforming it into a sequence of intermediate distributions, which can be more tractable and easier to sample from.

The annealing process gradually transforms the posterior distribution by introducing a temperature parameter $\beta$. By annealing from $\beta = 0$ to $\beta = 1$, we move from an initial distribution, where the posterior is approximated by a simpler distribution to the target posterior distribution itself. The key idea behind annealing is it allows for a smoother exploration of the posterior space. At each intermediate distribution, we can use importance sampling to estimate the evidence by sampling from the proposal distribution and reweighting the samples using the ratios of the target and proposal distributions.

As the annealing process progresses, the samples from the proposal distribution gradually become more representative of the target distribution. This means that the exploration of the posterior space is not limited to a specific region but covers a wider range of possible configurations of the latent variables. (Due to limited space, please refer to the text below ...)

The benefit of this exploration is that it allows for a more accurate estimation of the evidence, which corresponds to a tighter lower bound in the variational learning framework. By gradually annealing the temperature and exploring different distributions, SG-AIS can capture more complex structures in the posterior distribution, leading to better variational approximations in complex data and high-dimensional spaces. 

When choosing an appropriate annealing schedule for Stochastic Gradient Annealed Importance Sampling (SG-AIS), there are several trade-offs and considerations to keep in mind:

1. **Computational Efficiency**: The annealing schedule should be carefully designed to balance the computational resources required for estimating the evidence. Too many bridging densities can lead to excessive computational burden, while too few densities may result in less accurate estimates.

2. **Exploration vs Exploitation**: The annealing schedule should strike a balance between exploration and exploitation of the posterior distribution. An aggressive schedule that moves quickly from the base distribution to the posterior may lead to exploration limitations, while a slow schedule may lead to insufficient exploration and inefficiency.

3. **Smoothness of Transition**: The annealing schedule should ensure a smooth transition between bridging densities. Abrupt changes in the densities can result in high-variance importance weights, which may lead to inaccurate estimates. Smooth transitions can be achieved by gradually adjusting the temperature or using appropriate interpolation functions.

4. **Base Distribution**: The choice of a suitable base distribution is critical for effective sampling in AIS. It should be computationally efficient to sample from and cover the entire support of the target distribution. The base distribution should strike a balance between simplicity and representativeness.

Reviewer comment (Experiments):
The paper mentions that experimental results demonstrate improved performance on toy datasets and image datasets, but what about other types of data or real-world applications? Are there any known limitations or challenges when applying this approach to different domains? How about running experiments for bigger datasets such as Imagenet?

Author response:
The paper indeed focused on demonstrating the improved performance of SG-AIS on toy datasets and image datasets. However, when applying this approach to other types of data or real-world applications, there may be some limitations or challenges to consider.

One potential limitation could be the scalability of the method. As the size of the dataset increases, the computational resources required for estimating the evidence using SG-AIS may become more demanding. This is particularly true for large-scale datasets such as ImageNet, which contain millions of images. Running experiments on such massive datasets might pose challenges in terms of computational efficiency and memory requirements. 

We would like to emphasize that GPLVM indeed did not utilize ImageNet as a precedent dataset. Given that ImageNet involves higher-dimensional data, it may be more appropriate to combine GPLVM with other deep learning tools, such as convolutional neural networks (CNNs) and transformers. We are currently exploring broader application scenarios to incorporate these tools effectively and leave room for future work. Thank you for bringing up this point, as it is important to consider the specific requirements and complexities of high-dimensional datasets like ImageNet when exploring the applicability of SG-AIS.

Additionally, the annealing schedule plays a crucial role in the exploration of the posterior distribution. Designing an appropriate annealing schedule may require domain knowledge or trial and error experimentation. It might be necessary to tune the schedule to ensure a balance between exploration and exploitation, as well as a smooth transition between bridging densities. 

Regarding the applicability of SG-AIS in real-world applications, its performance may depend on the specific characteristics and requirements of the domain. Different datasets and applications may exhibit unique challenges, such as data sparsity, high dimensionality, or non-linear relationships, which could affect the effectiveness of SG-AIS. Evaluating the performance of SG-AIS in different domains and addressing these challenges would require further experimentation and investigation.

In summary, further research and experimentation are needed to assess its performance and address these challenges in real-world applications.