Paper: Near-optimal algorithms for private estimation and sequential testing of collision probability (ICLR-2024)

Reviewer comment (Novelty):
Poor contextualization of the results in comparison to previous work. Why focus on additive approximation? What are precisely the best known results?

Author response:
**Regarding additive error guarantees:** Butucea and Issartel (2021) and Bravo-Hermsdorff et al (2022), the closest previous work to ours, also proved additive error guarantees for privately estimating collision probability. The reviewer is correct that when no element of the domain has probability higher than $\epsilon$ then outputting zero satisfies an $\epsilon$ additive error guarantee. But determining whether this is the case is non-trivial.

Reviewer comment (Theory):
The paper does not show whether the studied problem is provably easier than heavy hitters in terms of sample complexity.

Author response:
**Regarding the difficulty of the problem:** In Appendix K of our latest revision we give a proof that our private collision probability estimator has significantly better sample complexity than the alternate approach suggested by the reviewer (and tested in our experiments) of reducing the problem to private distribution estimation.

Reviewer comment (Experiments):
The experiments are on simple uniform and power law distributions rather than specific datasets that would help demonstrate the applications of collisions probability estimation.

Author response:
**Regarding experiments on synthetic distributions:** Since collision probability is invariant to any reordering of the domain elements, we may assume without loss of generality that for any distribution $\mathbf{p}$, the probabilities $p_i$ decrease with $i$. Making this assumption clarifies that the only possible difference between real and synthetic distributions is the rate of decay of the $p_i$’s. In our original submission, we explored various decays by testing on both uniform and power law distributions (where $p_i \propto 1/i$). In our latest revision, we added experiments on the exponential distribution (where $p_i \propto \exp(-i)$). See Appendix J of the latest revision.

Reviewer comment (Theory):
I'm not sure I fully buy the claim that $\varepsilon$ is not necessary as an input to the collisions probability tester, since I think that at some point the algorithm does actually need to output something rather than run on indefinitely, even though the technical statements show a "for-all" guarantee that shows it would eventually reject for all values of $\varepsilon>0$. Perhaps the authors can address this point during the discussion phase.

Author response:
**Regarding whether $\epsilon$ can be unspecified for sequential testing:** We agree that in practice any testing algorithm must eventually be stopped, for example, when the sampling budget is exhausted. However, our theoretical guarantee ensures that the algorithm will not stop itself when the null hypothesis is true, and will eventually stop itself when the null hypothesis is false, without the user needing to know the value of $\epsilon$. Thus if the algorithm is prematurely stopped for practical reasons, the test will be inconclusive, but it will not produce a wrong answer (with high probability).

Reviewer comment (Theory):
Unless I'm missing something, the claim that the guarantee of Theorem 2 improves upon the prior result of Bravo-Hemsdorff et al. might be a bit misleading since their result holds for $\beta=0$ whereas Theorem 2 does not. While I'm not an expert, one expects that enforcing $\beta=0$ leads to different or incomparable technical challenges. However, the lower bound in Theorem 3 gives some evidence that the Bravo-Hemsdorff et al. result may be improved.

Author response:
**Regarding the approximate differential privacy parameter $\beta > 0$ instead of $\beta = 0$:** The DP literature contains many examples of mechanisms that increase $\beta$ from zero to a positive value, while improving the dependence on some other parameter. A classic example is the Gaussian mechanism, which has $\beta > 0$, but also improves performance for multiple queries relative to the Laplace mechanism, which has $\beta = 0$. See https://desfontain.es/privacy/gaussian-noise.html for a readable discussion.

Reviewer comment (Theory):
I might be missing some subtlety, but can one obtain the sequential testing guarantees using the existing results of Daskalakis and Kawase [ESA'17]? Namely, given a stream of samples $X_1,X_2,\ldots\in [k]$ from some fixed distribution, consider $Z_1,\ldots\in \{0,1\}$ with $Z_t = \mathbf{1}[X_{2t-1}=X_{2t}]$. Then the $Z_t$ are independent coin flips with bias $p$ where $p$ is the collision probability of the distribution. The two cases are then that $p=c_0$ or $\vert p-c_0\vert=\varepsilon>0$. My understanding is that Daskalakis and Kawase [Theorem 2, ESA'17] have shown that a natural sequential testing rule that, given a candidate $c_0$ and the stream $Z_1,Z_2,\ldots$, proceeds by testing whether this bias is close or far to $c_0$ with dyadically decreasing thresholds will properly reject after $\Theta(\log(\log(1/\varepsilon)/\delta)/\varepsilon^2)$ samples of the $Z$ with probability at least $1-\delta$ or never reject with probability at least $1-\delta$ in the good case.

Author response:
**Regarding a comparison to Daskalakis and Kawase (2017):** Their Algorithm 1 has two significant disadvantages relative to our Algorithm 2 that lead to much higher sample complexities in practice. First, their approach is based on a simple "doubling trick": They repeatedly invoke a non-sequential testing algorithm on subsequences of samples with successively smaller values of the error tolerance $\epsilon$, and stop when the testing algorithm rejects. This is a wasteful use of samples compared to our approach, as stopping cannot occur within a subsequence, and everything learned from previous subsequences is discarded. Second, applying their approach to collision probability testing requires partitioning the $n$ samples into $\frac{n}{2}$ disjoint pairs, so that the observed collisions are independent of each other. By contrast, our approach uses observed collisions among all $\binom{n}{2}$ pairs of samples to estimate collision probability, which significantly complicates the theoretical analysis, but leads to much better performance in practice (our other contribution, an improved mechanism for private collision probability estimation, also uses all $\binom{n}{2}$ pairs of samples to form estimates, while previous work used $\frac{n}{2}$ pairs).

We empirically compared the two algorithms, with the results shown in Appendix J of our latest revision. Our experiments demonstrate that as the null hypothesis approaches the true collision probability, the sample complexity of Daskalakis and Kawase's algorithm is much larger than ours. We will include this discussion and experiments in the final revision of our paper, and we thank the reviewer for raising this issue.

Reviewer comment (Theory):
Relatedly, Theorem 6 is nontrivial compared to the problem of testing (sequentially or not) the bias of a coin from independent samples since one gets strictly more information in this case. One suspects that the extra $\log\log$ term is inherent, but this does not appear to follow from the lower bounds of Daskalakis and Kawase since one gets more information in this setting (namely, the actual values in $[k]$). Closing this gap would be quite interesting.

Author response:
**Regarding a comparison to Daskalakis and Kawase (2017):** Their Algorithm 1 has two significant disadvantages relative to our Algorithm 2 that lead to much higher sample complexities in practice. First, their approach is based on a simple "doubling trick": They repeatedly invoke a non-sequential testing algorithm on subsequences of samples with successively smaller values of the error tolerance $\epsilon$, and stop when the testing algorithm rejects. This is a wasteful use of samples compared to our approach, as stopping cannot occur within a subsequence, and everything learned from previous subsequences is discarded. Second, applying their approach to collision probability testing requires partitioning the $n$ samples into $\frac{n}{2}$ disjoint pairs, so that the observed collisions are independent of each other. By contrast, our approach uses observed collisions among all $\binom{n}{2}$ pairs of samples to estimate collision probability, which significantly complicates the theoretical analysis, but leads to much better performance in practice (our other contribution, an improved mechanism for private collision probability estimation, also uses all $\binom{n}{2}$ pairs of samples to form estimates, while previous work used $\frac{n}{2}$ pairs).

We empirically compared the two algorithms, with the results shown in Appendix J of our latest revision. Our experiments demonstrate that as the null hypothesis approaches the true collision probability, the sample complexity of Daskalakis and Kawase's algorithm is much larger than ours. We will include this discussion and experiments in the final revision of our paper, and we thank the reviewer for raising this issue.