Paper: Improving Gradient-guided Nested Sampling for Posterior Inference (ICLR-2024)

Reviewer comment (Novelty):
Novelty is limited to a combination of existing methods.

Author response:
We have tried to highlight more clearly the original contributions from this paper: Firstly, we combine existing ideas that have never been combined before, resulting in a performance in sampling tasks that beats all existing state-of-the-art methods. Secondly, we use Nested Sampling with GFlowNets to generate a large number of samples from the target distribution. As the reviewer highlighted, we believe that this opens the door to many interesting research directions.

Reviewer comment (Evaluation):
In Figure 1, instead of plotting the error in the estimate of log Z versus the number of dimensions, how does the error grow versus the number of likelihood evaluations?

Author response:
We have added a plot of log Z vs. the number of likelihood evaluations in a separate appendix (Appendix J). GGNS needs fewer likelihood evaluations to reach the same error.

Reviewer comment (Evaluation):
In Figure 1, the error bar (i.e., standard deviation across 10 runs) are much bigger for GGNS compared with the baselines, does it mean it is trading off variance for bias?

Author response:
Interesting point. This is true inasmuch as the other methods do seem to have less variance while being more biased. At the same time, this is not an instance of the typical bias/variance tradeoff in probabilistic modeling: the other methods are biased because they are not correctly sampling the space (e.g., by missing modes, leading to an easier sampling problem). Therefore, it is not a matter of **more** model complexity or less regularization leading to less variance, but a matter of the **simpler** sampling methods being inaccurate.

Reviewer comment (Evaluation):
In Table 1, all methods except GGNS are under-biased for the Gaussian mixture example, is it just by chance?

Author response:
As far as we understand, the reason other methods underestimate the normalization constant for the Gaussian mixture is that they are likely to have missed, or in some way underestimated, a mode. This leads to these methods thinking that there is less total posterior mass and, therefore, that logZ is smaller. We have added a comment about this in the table caption.

Reviewer comment (Experiments):
In the 'Many Wells' experiment (Figure 4), are you comparing against FAB or FAB with buffer (which should give better results than FAB)?

Author response:
This is FAB with buffer, we have now clarified this in the text, thank you for pointing this out.

Reviewer comment (Presentation):
Formatting under Figure 1 needs to be modified.

Author response:
Thank you; these have been fixed.

Reviewer comment (Writing):
Page 7, "due to the high its high dimensionality" -> "due to its high dimensionality".

Author response:
Thank you; these have been fixed.

Reviewer comment (Writing):
In the introduction section, the authors list 4 differences from prior work. Conceptually I am bit confused about how each of these 4 parts work with each other. I would hope that the authors can elaborate a bit.

Author response:
To address the first point, we have added an appendix (Appendix H) with a complete GGNS algorithm. We have also added an ablation study (Appendix G) to analyse how our different improvements affect the algorithm’s performance.

Reviewer comment (Presentation):
A minor formatting issue: the caption of Figure 1 got cluttered and needs to be fixed later.

Author response:
We have also corrected the formatting issue with the caption of Figure 1.

Reviewer comment (Reproducibility):
My primary concern with the paper is the lack of clarity regarding the proposed method. While the contributions section seems to align with the methods, it primarily outlines algorithmic tweaks to existing methods. These minor modifications collectively seem to yield an impact. However, the paper does not showcase the final algorithm that encapsulates these changes; I only found references to current algorithms in the appendix. This absence of a clear differentiation in algorithmic terms hinders my ability to give a higher score, even though the experiments are commendable.

Author response:
We appreciate the difficulty in understanding how all the components of the algorithm come together. We have now added an appendix with a concrete GGNS algorithm (Appendix H).

Reviewer comment (Experiments):
I didn’t understand what was the advantage of the method for section 5. It seems a nice application that you can apply to generative flow networks, but what have we gained in training of the drift with your method, and is there a baseline we can compare your method against?

Author response:
In summary, Nested Sampling provides only a limited number of samples, but by training a GFlowNet, we can get an arbitrarily large number of samples by amortization.

GFlowNets are off-policy deep reinforcement learning algorithms. They are trained to sample a target density, but work best when the training is guided by a dataset of meaningful ground truth samples (to aid mode discovery), which need not be large or unbiased. The aim of Section 5 is thus twofold: we show that
- By using samples from GGNS in the training policy of a GFlowNet, we promote rapid mode discovery and convergence;
- After the GFlowNet has been trained (with GGNS guidance), we are able to use the trained model to obtain nearly-unbiased samples in a fixed number of sampling steps.

Reviewer comment (Presentation):
On page 5 at the top, there is some overlapping text below Figure 1.

Author response:
We have corrected the formatting error.

Reviewer comment (Reproducibility):
I don’t really understand how you took advantage of ‘GPU interoperability’.

Author response:
We have tried to clarify this point. The point here is that while Nested Samping algorithms had been written in differentiable programming languages before, none of those implementations took advantage of the gradient of the likelihood – their main advantage over implementations in non-differentiable programming languages arises from GPU interoperability. Therefore, what we meant to say (and we have tried to clarify), is that we do not take advantage only of GPU interoperability, but also of efficient parallel gradient computation.

Reviewer comment (Evaluation):
I don’t quite understand why the error bars reduce with the number of dimensions in Figure 3.

Author response:
This is a very good question, which we had not fully explored in the original submission. The reason for this is that $\mathcal{D}_{\rm KL} (\mathcal{P} | \Pi)$ decreases as we increase the dimensionality, as we show in appendix F.2, meaning that the information gain when going from prior to posterior goes down. This could occur because of the cancellations between sine and cosine times, as the number of terms in the sum increases.

Reviewer comment (Reproducibility):
While the paper discussed how to implement nested sampling, the exact details of the implementations are not self-contained within the paper. Therefore, it is difficult to actually grasp how the paper is proposing to implement these parts. The paper should be self-contained in terms of details, explaining algorithmic details, providing insight, and contrasting with previous implementations, rather than relying mostly on textual explanation that lacks technical preciseness.

Author response:
First, we appreciate the comment about the lack of details of the algorithm. Therefore, we have added an Appendix H with the details of our implementation. We hope this will add clarity and transparency to the paper.

Reviewer comment (Reproducibility):
Section 3 "Adaptive Time Step Control": The paper states "This time step is adjusted dynamically ... increase or decrease ...", but does not explain exactly how it is adjusted.

Author response:
The details of how adaptive time step happens are now in lines 21-24 of Algorithm 3.

Reviewer comment (Reproducibility):
Section 3 "Trajectory preservation": The statement "select a new live point at random from the stored trajectories" is vague. Is this selection uniformly at random or weighted resampling as in typical Hamiltonian Monte Carlo implementations [2,3]?

Author response:
By trajectory, we mean the steps of the Markov chain, we have clarified it in a footnote in Section 3. The points are sampled fully randomly, as we need to sample uniformly from the prior. We have also clarified this.

Reviewer comment (Writing):
Section 3 "Differentiable Programming": The paper does not explain how differentiable programming is used here and why it helps.

Author response:
Differentiable programming is key here, in that our method requires calculating the gradient of the likelihood. Therefore, it will only work for likelihoods for which we can compute gradients

Reviewer comment (Theory):
Section 3 "Mode Collapse Mitigation": The claim about "... preventing them from converging prematurely to a single model" is made without theoretical or empirical evidence.

Author response:
We have added empirical evidence for this in our ablation study (Appendix G.3).

Reviewer comment (Theory):
Section 3 "Robust Termination Criterion": The statement "terminate ... has decreased by a predetermined fraction from its maximum value" lacks explanation of the theoretical principle behind this termination criterion.

Author response:
We have added a more detailed motivation for our termination criterion in appendix I, as well as empirical evidence of it working better than the commonly used in in appendix G.4.

Reviewer comment (Experiments):
The experiments are inconclusive in terms of experimental design and choice of baselines. The paper claims that it significantly improves performance in high-dimensional settings, but none of the experiments are necessarily high-dimensional by today's standards (e.g., Section 4.1 Figure 1 only goes up to 128 dimensions, versus tens of thousands in [4]). If scalability only holds among nested sampling algorithms, the significance is questionable since more scalable alternatives exist.

Author response:
We tested the algorithm up to $d=256$, which is a higher dimensionality that has ever been used in nested sampling. In such high dimensions, simpler MCMC-based samplers (HMC, NUTS) have traditionally been used. Our proposed approach is the first to bring the benefits of nested sampling – namely, the ability to reliably sample multimodal distributions and calculate the evidence – to such high-dimensional spaces.

Reviewer comment (Experiments):
Some baselines are unclear. In Section 4.2, HMC is used as a baseline, but HMC alone does not produce a log-evidence estimate unless special tricks are used. The paper does not explain how HMC was used to produce a log-evidence. Similarly, the citation of Halton (1962) for sequential Monte Carlo seems unrelated. The intended citation is unclear, as sequential Monte Carlo is usually attributed to other works [7,8,9].

Author response:
This result is taken from [Zhang and Chen, 2022], which presumably used the harmonic mean estimator to calculate the evidence given the obtained samples.

We thank the reviewer pointing out this oversight, we have now added these citations to the paper.

Reviewer comment (Experiments):
The baselines are insufficient to judge performance. In statistics, log-evidence estimation is often done using thermodynamic integration or bridge sampling [10], which are not included. If sequential Monte Carlo as in [6] was intended, more implementation details are needed due to its sensitivity to implementation details.

Author response:
The estimation of the log-evidence is only one of the benchmarks we use, and it is commonly used to benchmark sampling methods. We used this benchmark because it has also been used in [Zhang and Chen, 2022] and [Lahlou et al., 2023]. 

We are using the result taken from [Zhang and Chen, 2022]. We have highlighted their SMC implementation details in the paper, for clarity.

Reviewer comment (Experiments):
The authors did not use the same set of baselines for all problems. For example, comparable nested sampling methods are only used in Section 4.1.

Author response:
The logic here was that Section 4.1 shows our nested sampling implementation outperforms others, so the next sections focus on comparing us to different methods. The amount of available methods is so large, that it felt very impractical to compare with every method on every experiment.

Reviewer comment (Writing):
Section 1 first paragraph: Hamiltonian Monte Carlo was initially developed by Duane et al. [11].

Author response:
We thank the referee for pointing this out. We have added the citation.

Reviewer comment (Novelty):
A similar reflective version of the HMC algorithm was developed by [12], although they did not consider the nested sampling setting.

Author response:
This is certainly a very relevant paper, which we were not aware of. We have now added a citation to the paper, as well as placed it in the context of our work.

Reviewer comment (Writing):
Section 1 second paragraph: The reviewer did not understand the intention of the sentence "From the perspective of differentiable programming, less attention has been paid in recent years". It is unclear what context differentiable programming has to do with sampling here.

Author response:
What we meant here was that differentiable programming, and particularly differentiable likelihoods, has been used for sampling methods such as HMC, SMC, as well as deep learning methods. But not for nested sampling.

Reviewer comment (Novelty):
The main contribution of the paper is a combination of different existing methods. The paper lacks originality and significance.

Author response:
We thank the reviewer for their feedback. We have tried to address all four points that were brought up:

1. We have tried to further clarify our main contributions. Please see response to all reviewers for a summary.

Reviewer comment (Reproducibility):
The reviewer cannot find the proposed GGNS algorithm. The authors need to give a concrete GGNS algorithm.

Author response:
We thank the reviewer for bringing up this point. We have now added an appendix (Appendix H) with a concrete GGNS algorithm, including all the details.

Reviewer comment (Theory):
There is no theoretical analysis regarding the GGNS. There is no theoretical guarantee for the advantages of the GGNS algorithm.

Author response:
We appreciate this point. To attempt to address it, we have tried to provide a justification for what is, in our opinion, the most important part of this work, which is the fact that each step only needs O(1) bounces; as this is what allows the algorithm to have a better scaling with the number of dimensions. This has been added to Section 3. In summary, while slice sampling only has information about one dimension per step (needing $O(n)$ steps to explore an $n$-dimensional space), when we use Hamiltonian slice sampling, the gradient provides information about all $n$ dimensions on each step. The $SO(n)$-equivariance of Hamiltonian slice sampling implies that exploration of a density concentrated around a low-dimensional manifold is less sensitive to the dimension of the ambient space. Please also see the answer to Reviewer skHM for intuition regarding this.

Reviewer comment (Experiments):
Since the method is a combination of different strategies, the authors need to conduct an abolition study on the strategies to validate their effectiveness.

Author response:
This is a very good point. We have added an Appendix G with a detailed ablation study. Please see the response to all reviewers for discussion.