Paper: Distance Estimation for High-Dimensional Distributions (ICLR-2024)

Reviewer comment (Theory):
Although there is good intuition for parts of the main result, I think the paper could benefit from additional algorithmic/analytic intuition rather than the full formal proofs. For example, I would have liked to see more details about how the tamed distribution could be accessed using subcube conditioning queries. I would have also liked to see intuition on where the $n^3$ and $1/\varepsilon^5$ factors come from.

Author response:
We have reorganized the exposition of the proof to make it more intuitive. Additionally, we have fleshed out the part about taming to make the role of SUBCOND clearer. Further, we have also added an informal description of SubToEval with the aim of making it clear how we arrived at $n^3$ and $\varepsilon^{-5}$.

Reviewer comment (Novelty):
It is not clear to me that the applications of distance estimation with subcube conditioning queries is well-suited for the particular learning theory community at ICLR.

Author response:
Q4. What are the additional applications of distance estimation with subcube conditioning queries?  
Ans. Distance estimation with subcube conditioning has applications in other fields, such as databases and probabilistic circuits, as subcube conditional queries are natural to these domains.

Reviewer comment (Experiments):
Although I do not think experiments are necessary for a paper with solid theoretical foundations, I think the experimental section of this paper is a bit unclear (see questions below).

Author response:
Q2. How was the scalable benchmark generated, and how are the real-world circuits represented by the Boolean formulas?  
Ans. As mentioned in the experimental evaluation, the scalable benchmarks are random 3-cnfs of varying numbers of variables. On the other hand, the real-world circuits are already available in CNF and were obtained from a set of publicly available benchmarks arising from sampling and counting tasks (https://zenodo.org/records/3793090). We have added details regarding the benchmarks to the revised version of the paper.  

Q3. How does the tradeoff between sample complexity and TVD look like for these datasets?  
Ans. We will analyze the sample complexity with respect to TVD and present it in a later revision.

Reviewer comment (Experiments):
Dimensions in your experiments are not high enough, and $\epsilon$ is too large.

Author response:
Regarding Experiments: 
There has been no prior work that has been able to scale to even ~40 dimensions for the case of constrained sampling, as constrained samplers tend to be slow. We would like to point out that the previous estimators would have required significantly more time and samples due to their exponential(in n) sample complexity.

We were not able to perform experiments with larger dimensions due to the lack of computing resources. Currently, every instance requires around 4*10^5 s = 5 days of computer time. 

As mentioned in the paper previously, there were no distance estimators that drew only poly(n) samples. In a later revision, we will add a discussion of the previous exponential sample algorithm to our related work section to place our work better.

Reviewer comment (Theory):
Is that possible to provide a high probability bound with respect to the number of queries? If not, could you provide the technical hardness?

Author response:
Q) Is it possible to provide a high probability bound with respect to the number of queries? If not, could you provide the technical hardness?
A) Yes, we can show a high-probability bound with the help of the median trick. We have updated the proof in the revised version to show the mentioned bound.

Reviewer comment (Presentation):
There needs to be an example in the introduction that describes scenarios in which the SUBCOND query is available. Personally, I have been very curious in such scenarios until I read the Application section.

Author response:
Q) There needs to be an example in the introduction that describes scenarios in which the SUBCOND query is available. Personally, I have been very curious in such scenarios until I read the Application section..

A) For our experiments we focus on samplers that sample satisfying assignments of a CNF, and \subcond is particularly well suited for this problem, as a CNF conditioned on a subcube is also a CNF. This enables us to recursively solve the problem as the subproblems are of the same type. As suggested, we have added the example from the application section to the introduction of our paper.

Reviewer comment (Theory):
Algorithm 1 returns the average of $1-p_i-q_i$. Should the definition of $Z$ in Lemma 1 be divided by $m$ as well?

Author response:
We thank the reviewer for their careful reading and helpful suggestions.  Z was indeed missing a factor of m, which we have corrected in the updated version.

Reviewer comment (Presentation):
The title promises too much compared to what the paper delivers. When most people think of high dimensional distributions, they don't have the Boolean hypercube in mind as the domain, they would think of $\mathbb{R}^n$. I would suggest adding "discrete distributions" to the title, and possibly mentioning the need for conditional samples.

Author response:
As suggested, we have changed the title to “Distance Estimation for High-Dimensional Discrete Distributions” to reflect our contributions more accurately.

Reviewer comment (Theory):
On a related note, the SUBCOND model is very natural in the discrete setting (for product domains). I wonder to what extent these techniques can extend to other domains, and what a reasonable analog of this model might be.

Author response:
Our contribution extends to the hypergrid setting $[d_1] \times [d_2] \times \ldots [d_n]$, where $d_i$ is any discrete set, and we will add this to a later revision. We note that in practice, reals are represented as discrete floats, making SUBCOND potentially useful in some settings with real dimensions.