On Scalable Testing of Samplers ∗†
Abstract
In this paper we study the problem of testing of constrained samplers over highdimensional distributions with ( ε, η, δ ) guarantees.
Samplers are increasingly used in a wide range of safety-critical ML applications, and hence the testing problem has gained importance.
For n -dimensional distributions, the existing state-of-the-art algorithm, Barbarik2 , has a worst case query complexity of exponential in n and hence is not ideal for use in practice.
Our primary contribution is an exponentially faster algorithm that has a query complexity linear in n and hence can easily scale to larger instances.
We demonstrate our claim by implementing our algorithm and then comparing it against Barbarik2 .
Our experiments on the samplers wUnigen and wSTS , ﬁnd that Barbarik3 requires 10 × fewer samples for wUnigen3 and 450 × fewer samples for wSTS as compared to Barbarik2 .
Introduction
The constrained sampling problem is to draw samples from high-dimensional distributions over constrained spaces.
A constrained sampler Q ( ϕ, w ) takes in a set of constraints ϕ : { 0 , 1 } n → { 0 , 1 } and a weight function w : { 0 , 1 } n → R > 0 , and returns a sample σ ∈ ϕ − 1 (1) with probability proportional to w ( σ ) .
Constrained sampling is a core primitive of many statistical inference methods used in ML, such as Sequential Monte Carlo[29], Markov Chain Monte Carlo(MCMC)[3, 9], Simulated
Annealing [4], and Variational Inference [24].
Sampling from real-world distributions is often computationally intractable, and hence, in practice, samplers are heuristical and lack theoretical guarantees.
For such samplers, it is an important problem to determine whether the sampled distribution is close to the desired distribution, and this problem is known as testing of samplers .
The problem was formalised in [14, 27] as follows: Given access to a target distribution P , a sampler Q ( ϕ, w ) , and three parameters ( ε, η, δ ) , with probability at least 1 − δ , return (1) Accept if d ∞ ( P , Q ( ϕ, w )) < ε , or (2) Reject if d TV ( P , Q ( ϕ, w )) >
η .
Here d TV is the total variation distance, d ∞ the multiplicative distance, and ε, η, and δ are parameters for closeness, farness and conﬁdence respectively.
Access to distribution P is via the DUAL oracle, and access to Q is via the PCOND and SAMP oracles (deﬁned in Section 2.1)
There is substantial interest in the testing problem due to the increasing use of ML systems in real-world applications where safety is essential, such as medicine [2], transportation [8, 25], and warfare [28].
For the ML systems that incorporate samplers, the typical testing approach has been to show the convergence of the sampler with the target distribution via empirical tests that rely on heuristics and do not provide any guarantees [19, 22, 31, 34].
In a recent work [27], a novel framework, called Barbarik2 , was proposed that could test a given sampler while providing ( ε, η, δ ) guarantees, using ˜ O (cid:16) tilt ( P ) 2 η ( η − 3 ε ) 3 (cid:17) queries, where tilt ( P ) :=
[Equation]
P (
[Equation]
) P ( [Equation] 2 ) for P ( σ 2 ) > 0 .
Since the tilt ( P ) can take arbitrary values, we observe that the query complexity can be prohibitively large 3 .
On the other hand, the best known lower bound for the problem, derived from [30], is ˜Ω (cid:18) √ n/ log( n ) η(cid:19) .
In this work, we take a step towards bridging this gap with our algorithm, Barbarik3 , that has a query complexity of ˜ O (cid:16) √ n log n ( η − 11 .
6 ε ) η 3 + nη 2 (cid:17) , representing an exponential improvement over the state of the art.
To be of any real value, testing tools must be able to scale to larger instances.
In the case of constrained samplers, the only existing testing tool, Barbarik2 , is not scalable owing to its query complexity.
The lack of scalability is illustrated by the following fact: product distributions are the simplest possible constrained distributions, and given a union of two n -dimensional product distributions, Barbarik requires more than 10 8 queries for n > 30 .
On the other hand, the query complexity of Barbarik scales linearly with n , the number of dimensions, thus making it more appropriate for practical use.
We implement Barbarik3 and compare it against Barbarik2 to determine their relative performance.
In our experiments, we consider two sets of problems, (1) constrained sampling benchmarks, (2) scalable benchmarks and two constrained samplers wSTS and wUnigen3 .
We found that to complete the test Barbarik3 required at least 450 × fewer samples from wSTS and 10 × fewer samples from wUnigen3 as compared to Barbarik2 .
Moreover, Barbarik3 terminates with a result on at least 3 × more benchmarks than Barbarik2 in each experiment.
Our contributions can be summarized as follows:
For the problem of testing of samplers, we provide an exponential improvement in query complexity over the current state of the art test Barbarik2 .
Our test, Barbarik3 , makes a total of ˜ O (cid:16) √ n log n ( η − 11 .
6 ε ) η 3 + nη 2 (cid:17) queries, where ˜ O hides polylog factors of ε, η and δ .
We present an extensive empirical evaluation of Barbarik3 and contrast it with Barbarik2 .
The results indicate that Barbarik3 requires far fewer samples and terminates on more benchmarks when compared to Barbarik2 .
We deﬁne the notation and discuss related work in Section 2.
We then present the main contribution of the paper, the test Barbarik3 , and its proof of correctness in Section 3.
We present our experimental ﬁndings in Section 4 and then we conclude the paper and discuss some open problems in Section 5.
Due to space constraints, we defer some proofs and the full experimental results to the supplementary
Section A and B respectively.
Notation and preliminaries
Probability distributions In this paper we deal with samplers that sample from discrete probability distributions over high-dimensional spaces.
We consider the sample space to be the n -dimensional
Boolean hypercube { 0 , 1 } n .
A constrained sampler Q takes in a set of constraints ϕ : { 0 , 1 } n → { 0 , 1 } and a weight function w : { 0 , 1 } n → R > 0 , and samples from the distribution Q ( ϕ, w ) deﬁned as
Q ( ϕ, w )( σ ) = (cid:26) w ( σ ) / w ( ϕ
[Equation]
[Equation]
0) , where w ( ϕ ) = (cid:80)
[Equation]
[Equation]
w ( σ ) .
To improve readability, we use Q to refer to the distribution Q ( ϕ, w ) .
For an element i , D ( i ) denotes it’s probability in distribution D
and i ∼ D represents that i is sampled from D .
For any non-empty set S ⊆ { 0 , 1 } n ,
D S is the distribution D conditioned on set S , and D ( S ) is the probability of S in D i.e., D ( S ) =
(cid:80) i ∈ S D ( i ) .
The total variation (TV) distance of two probability distributions D 1 and D 2 is deﬁned as: d TV ( D 1 , D 2 ) =
12 (cid:80) i ∈{ 0 , 1 } n |D 1 ( i ) − D 2 ( i ) | .
For S ⊆ { 0 , 1 } n , we deﬁne d TV ( S )
( D 1 , D 2 )
= 12 (cid:80) i ∈ S |D 1 ( i ) − D 2 ( i ) | .
The multiplicative distance of D 2 from D 1 is deﬁned as: d ∞ ( D 1 , D 2 ) =
max i ∈{ 0 , 1 } n |D 2 ( i ) /
D 1 ( i ) − 1 | .
The two notions of distance obey the identity: 2 d TV ( D 1 , D 2 ) ≤ d ∞ ( D 1 , D 2 ) .
In the rest of the paper, E [ v ] represents the expectation of random variable v and [ k ] represents the set { 1 , 2 . . .
, k } .
[Figure]
[Equation]
Tools used in the analysis
Proposition 1 (Hoeffding) .
For independent 0-1 random variables X i , X
= (cid:80) ki
=1 X i , and t ≥ 0 ,
Pr( X − E X > t ) ≤ exp (cid:0) − 2 t 2 k (cid:1) and Pr( E X − X > t ) ≤ exp (cid:0) − 2 t 2 k (cid:1)
Proposition 2 (Chebyshev) .
Given bounded r.v. X , we have Pr( | X − E
[ X ] | < E [ X ]) >
E [ X ] 2 E [ X 2 ]
Proposition 3.
Given distributions D 1 and D 2 supported on { 0 , 1 } n , and a set S ⊆ { 0 , 1 } n , (cid:88)
[Equation]
The proof can be found in the Appendix [-A.
-]A.
If we are given samples { s 1 , s 2 , . . .
, s n } from a distribution D over [ k ] , then the empirical distribution (cid:98) D is deﬁned to be (cid:98) D ( i ) =
1 n k (cid:80) j = 1 { s j = i } .
Proposition 4 (See [11] for a simple proof) .
Suppose D is a distribution over [ k ] , and (cid:98) D is constructed using max (cid:16) kη 2 ,
2ln(2 /δ ) η(cid:17) samples from D .
Then d TV ( D , (cid:98) D ) ≤
η with probability at least 1 − δ .
2.1 Testing with the help of oracles
In distribution testing, we are given samples from an unknown distribution P over a large support { 0 , 1 } n , and the task is to test whether P satisﬁes some property of interest.
One of the important properties we care about is whether P is close to another distribution Q , and this subﬁeld of testing is known as closeness testing .
It was shown by Valiant and Batu et al.
that the ability to draw samples from P and Q is not powerful enough, as at least Ω(2 2 n/ 3 ) samples are required to provide any sort of probabilistic guarantee for closeness testing.
Since n is usually large, it was desirable to ﬁnd tests that could solve the closeness testing problem using polynomially many samples in n .
Motivated by the above requirement, Canonne et al.
and Chakraborty et al. introduced the conditional sampling oracle ( COND ), that is a more powerful way to access distributions.
A COND oracle for distribution D over { 0 , 1 } n takes as input a set S ⊆ { 0 , 1 } n with D ( S ) > 0 , and returns a sample i ∈ S with probability D ( i ) / D ( S ) .
It has been shown that the use of the COND oracle, and its variants, drastically reduces the sample complexity of many tasks in distribution testing [1, 21, 12, 15, 6, 26, 7, 17, 13, 30] (see [10] for an extensive survey).
In this paper, we consider the pair-conditioning ( PCOND ) oracle, which is a special case of the COND oracle with the restriction that | S | = 2 i.e., the size of the conditioning set has to be two.
To engineer practical PCOND oracle access into constrained samplers, we use the chain formula construction introduced in [14].
With the same goal of designing tests with polynomial sample complexity, a different kind of oracle, known as the DUAL oracle, was proposed by Canonne et al..
The DUAL oracle allows one to sample from a given distribution and also query the distribution for the probability of arbitrary elements of the support.
Tractable DUAL oracle access is supported by a number of distribution representations, such as the fragments of probabilistic circuits (PC) that support the EVI query [18].
In our experimental evaluation, we use distributions from one such fragment: weighted d - DNNF s. Weighted d - DNNF s are a class of arithmetic circuits with properties that enable DUAL oracle access in time linear in the size of the circuit [16, 23].
Barbarik3 : an algorithm for testing samplers
We start by providing a brief overview of our testing algorithm before providing the full analysis.
3.1 Algorithm outline
The pseudocode of Barbarik3 is given in Algorithm 1.
We adapt the deﬁnition of bucketing of distributions from [30] for use in our analysis.
Deﬁnition 1.
For a given k ∈ N > 0 , the bucketing of { 0 , 1 } n with respect to P is deﬁned as follows:
For 1 ≤ i ≤ k , let S i
= { b : 2 − i < P ( b ) ≤ 2 − i +1 } and let S
= { 0 , 1 } n \ (cid:83) i ∈
[ k ] S i .
Given 1: k ← n + (cid:100) log 2 (100 /η ) (cid:101) 2: for i = 1 to k do 3: S i = { b : 2 − i < P ( b ) ≤ 2 − i +1 } 4: S 0 = { 0 , 1 } n \ (cid:83) i ∈ [ k ] S i 5: B P is the distribution over [ k ] ∪ { 0 } where we sample i ∼ B P if we sample j ∼ P and j ∈ S i 6: B Q is the distribution over [ k ] ∪ { 0 } where we sample i ∼ B Q if we sample j ∼ Q and j ∈ S i 7: θ ← η/8: (cid:98) d ← OutBucket ( B P , B Q , k, θ, δ/ 2) 9: if (cid:98) d > ε/ 2 + θ then 10:
Return Reject 11: ε 2 ← (cid:98) d
+ θ 12: Return InBucket ( P , Q , k, ε, ε 2 , η, δ/ 2) any distribution D over { 0 , 1 } n , we deﬁne a distribution B D over [ k ] ∪ { 0 } as: for 0 ≤ i ≤ k ,
B D ( i ) =
D ( S i ) .
We call B D the bucket distribution of D and S i the i th bucket.
Barbarik3 takes as input two distributions P and Q deﬁned over the support { 0 , 1 } n , along with the parameters for closeness( ε ), farness( η ), and conﬁdence( δ ).
On Line 1, Barbarik3 computes the value of k using η and the number of dimensions n .
Then, using DUAL access to P , and SAMP access to
Q , Barbarik3 creates bucket distributions B P and B Q as in Defn.
1, in the following way: To sample from {+B P , Barbarik
+}
ﬁrst draws a sample j ∼ P , then using the DUAL oracle, determines the value of P ( j ) .
Then, if j lies in the
i th bucket i.e., 2 −
i < P ( j ) ≤ 2 − i +1 , the algorithm takes sample i as the sample from B P .
Similarly, to draw a sample from B Q , Barbarik3 draws a sample j ∼ Q and then, using the DUAL oracle to ﬁnd P ( j ) , ﬁnds i such that j lies in the i th bucket, and then uses i as the sample.
Barbarik3 then calls two subroutines, OutBucket (Section 3.4) and InBucket (Section 3.3).
The
OutBucket subroutine returns an θ -multiplicative estimate of the TV distance between B P and B Q , the two bucket distributions of P and Q , with an error of at most δ/ 2 .
If it is found on Line 9 that the estimate (cid:98) d is greater than ε/ 2 + θ , we know that d TV ( P , Q ) > ε/ 2 and also that d ∞ ( P , Q ) > ε , and hence the algorithm returns Reject .
Otherwise, the algorithm calls the InBucket subroutine.
Now suppose that d TV ( P , Q ) ≥ η .
Then, for ε 2 (Line 11), it is either the case that d TV ( B P , B Q ) >
ε 2 or else d TV ( B P , B Q ) ≤ ε 2 .
In the former case, the algorithm returns Reject on Line 10, and in the latter case the InBucket subroutine returns Reject .
In both cases, the failure probability is at most δ/ 2 .
Thus Barbarik3 returns Reject on given η -far input distributions with probability at least 1 − δ .
We will now prove the following theorem:
Theorem 1.
{+Barbarik
+} ( P , Q , η, ε, δ ) takes in distributions P and Q deﬁned over { 0 , 1 } n , and parameters η ∈ (0 , 1] , ε ∈ [0 , η/ 11 . 6) and δ ∈ (0 , 1 / 2] .
Barbarik3 has DUAL access to P , and
PCOND + SAMP access to Q .
With probability at least 1 − δ , Barbarik3 returns
• Accept if d ∞ ( P , Q ) ≤ ε • Reject if d TV ( P , Q ) >
η Barbarik3 has query complexity ˜ O (cid:16) √ n log( n ) η 3 ( η − 11 .
6 ε ) + nη 2 (cid:17) , where ˜ O hides polylog factors of ε, η and δ .
3.2 Lower bound
The lower bound comes from the paper of Narayanan [30], where it appears in Theorem 1.6.
Phrased in the jargon of our paper, the lower bound states that distinguishing between d TV ( P , Q ) > η and d ∞ ( P , Q ) = 0 requires ˜Ω( (cid:112) n/ log( n ) /η 2 ) samples.
Note that the lower bound is shown on a special case ( ε = 0 ) of our problem.
Hence the lower bound applies to our problem as well.
Furthermore, the lower bound is shown for the case where distribution P provides full access, i.e., the algorithm can make arbitrary queries to P .
This is a stronger access model than DUAL .
Since the lower bound is for a stronger access model, it extends to our problem as well.
3.3 The InBucket subroutine
In this section, we present the InBucket subroutine, whose behavior is stated in the following lemma.
Lemma 1.
InBucket ( P , Q , k, ε, ε 2 , η, δ ) takes as input two distributions P , Q , an integer k and parameters ε, ε 2 , η, δ .
If d ∞ ( P , Q ) ≤ ε , InBucket returns Accept .
If d TV ( P , Q ) ≥ η and d TV ( B P , B Q ) < ε 2 , then InBucket returns Reject .
InBucket errs with probability at most δ .
[Table] 5: for t iterations do
[-25 ε 2 − 2 ε/ (1 − ε )) / 1 . 
-][Table] [Table]
m samples
[Table]
P
[Table]
S
[Table]
is deﬁned in Defn.
[Table]
m [Table] [Table] 10: for all j ∈ [ k ] s.t. | Γ
[Table] P | , [Table] 0 do 11: p
[Table]
p is an [Table] sample
[Table]
the
[Table]
j [Table]
[Table] is an
[Table]
sample from the [Table] [Table] [Equation]
[Table]
[Table]
P ( q )(1+ α ) 15: r ← (cid:108) 2ln(4 mt/δ ) ( h − (cid:96) ) 2 (cid:109) 16: (cid:98) c
← Bias ( Q, p, q, r ) 17: if (cid:98) c ≤ ( h + (cid:96) ) /
2 then 18:
Return Reject 19: Return Accept
Algorithm 3 Bias ( Q , p,
) 1: if p and q are identical then 2:
Return 0 .3:
Γ Q { p,q } ← r samples from Q { p,q } 4: Return # of times p appears in Γ Q { p,q }
InBucket makes extensive use of the PCOND oracle access to Q via the Bias subroutine, which we describe in the following subsection.
The Bias subroutine The Bias subroutine takes in distribution Q , two elements p, q and a positive integer r .
Then, using the PCOND oracle, Bias draws r samples from the conditional distribution
Q { p,q } and returns the number of times it sees p in the r samples.
It can be seen that the returned value is an empirical estimate of
Q ( p ) Q ( p )+ Q ( q ) .
Let the estimate be (cid:99) c pq .
We use the Hoeffding bound in
Prop. 1, and the value of r from Line 15 of Alg.
(2) to show that:
[Equation]
Here t represents the number of iterations of the outer loop (Line 4), and m is the number of samples drawn from B P and B Q .
Together, there are at most mt pairs of samples that are passed to the
Bias oracle.
Since in each invocation of Bias , the probability of error is δ/ 4 mt , using the union bound we ﬁnd that the probability that all mt Bias calls return correctly is at least 1 − δ/ 4 and thus with probability at least 1 − δ/ 4 , the empirical estimate (cid:99) c pq is closer than ( h − (cid:96) ) / 4 to
Q ( p ) Q ( p )+ Q ( q ) .
Henceforth we assume:
[Equation]
3.3.1 The Accept case
In this section we will provide an analysis of the case when d ∞ ( P , Q ) < ε .
We will now state a proposition required for the remaining proofs, the proof of which we relegate to Appendix A.4.
Proposition 5.
Let P , Q be distributions and let p ∼ P and q ∼ Q .
Then,
If d ∞ ( P , Q ) < ε
then
Q
[Equation]
p
[Equation]
p
[Equation]
1 + 2 ε
−
ε
[Equation]
P
[Equation]
If d TV ( P , Q ) > ε 1 , then for 0 ≤ α < ε 1 , with probability at least ( d TV ( P , Q ) − α ) / 2 , Q ( p
[Equation]
p
[Equation]
P
[Equation]
p
[Equation]
P ( p )
[Equation]
1 + α
[Equation]
P [Equation] q ) From our assumption (1), we know that for all invocations of Bias , with probability at least 1 − δ/ 4 , (cid:12)(cid:12)(cid:12)(cid:99) c pq −
Q ( p ) Q ( p )+ Q ( q ) (cid:12)(cid:12)(cid:12) ≤ ( h − (cid:96) ) / 2 .
Using Prop. 5, and using the value of h given on Line 13, we can see that
Q ( p ) Q ( p )+ Q ( q ) > h .
From this we can observe that for all invocations of Bias , (cid:99) c pq > ( h + (cid:96) ) /and
the test does not return Reject in any iteration, hence eventually returning Accept .
Thus, in the case that d ∞ ( P , Q ) < ε , the InBucket subroutine returns Accept with probability at least 1 − δ/ 4 .
3.3.2 The Reject case
In this section we analyse the case when d TV ( P , Q ) ≥ η and d TV ( B P , B Q ) ≤ ε 2 and we will show that the algorithm returns Reject with probability at least 1 − δ .
For the purpose of the proof we will deﬁne a set of bad buckets Bad ⊆
[ k ] .
Note that bucket { 0 } is not in Bad .
Deﬁnition
[Equation]
[Equation]
Suppose we have an indicator variable X r,s constructed as follows: draw m samples from P and Q , and if the r th sample from P and the s th sample from Q both belong to some bucket b ∈ Bad , then
X r,s = 1 else X r,s = 0 .
Then,
[Equation]
The inequality is by the application of Prop. 3.
We analyse the expression for the expectation in the following lemma, the proof of which we relegate to Appendix [-A.
-]A.
Lemma 2.
[Equation]
Using Lemma 2 we immediately derive the fact that E [ X r,s ] > (cid:0) 0 .
99 η − 134 ε 2 − ε 1 (cid:1) 2 /K .
Let
X = (cid:80) r,s ∈ [ m ] X r,s .
Given m samples from P and Q , Pr( X ≥ 1) is the probability that there is at least one bucket in Bad that is sampled at least once each in both sets of samples.
Lemma 3.
Pr( X ≥ 1) > 1 /[-
-]
The proof can be found in Appendix A.3.
Henceforth we will condition on the the event that X ≥ 1 .
In such a case, we know that for some k ∈ Bad , there is a sample p ∼ P S k and a sample q ∼ Q S k .
Then for such a pair of samples ( p, q ) , and some α , Prop. 5 tells us that with probability at least ( d TV ( P , Q ) − α ) /
2 we have
[Equation]
Using the assumption made in (1), we immediately have that (cid:99) c pq ≤
Q ( p ) Q ( p )+ Q ( q ) + h − (cid:96) 2 .
But from
Prop.
we have that
Q ( p ) Q ( p )+ Q ( q ) < (cid:96) and hence (cid:99)
c pq < ( h + (cid:96) ) / 2 .
Since d TV ( P , Q ) ≥ ε 1 , we see that if X ≥ 1 , then with probability at least ( ε 1 − α ) / 2 , the iteration returns Reject .
Then, using Lemma 3 we see that in every iteration, with probability at least ( ε 1 − α ) / 10 , InBucket returns Reject .
There are t iterations, where t (line 4) is chosen such that the overall probability of the test returning Reject is at least 1 − δ/ 2 .
3.4 The OutBucket subroutine
The OutBucket subroutine takes as input two distributions D 1 , D 2 over k + 1 elements and two parameters θ and δ .
Then with probability at least 1 − δ , InBucket returns a θ -multiplicative estimate for d TV ( D 1 , D 2 ) .
The OutBucket starts by drawing max (cid:16) 4( k +1) θ, 8ln(4 /δ ) θ(cid:17) samples from the two distributions Dand D 2 , and constructs the empirical distributions (cid:99) D 1 and (cid:99) D 2 .
Then from Prop. 4, we know that with probability at least 1 − δ , both d TV ( D 1 , (cid:99) D 1 ) ≤ θ/ 2 and d TV ( D 2 , (cid:99) D 2 ) [Equation] θ/ 2 .
From the triangle inequality we have that,
[Equation]
and also that,
[Equation]
Thus with probability at least 1 − δ , the returned estimate d TV ( (cid:99) D 1 , (cid:99) D 2 ) satisﬁes | d TV ( (cid:99) D 1 , (cid:99) D 2 ) − d TV
( D 1 , D 2 ) | < θ .
Query and runtime complexity The number of queries made by OutBucket to P and Q is given by ˜ O (cid:16) nη 2 (cid:17) , where ˜ O hides polylog factors of ε, η and δ .
The number of queries required by InBucket is given by mtr .
Bounding the terms individually, we see that m =
˜ O (cid:16) √ n η − 11 .
6 ε (cid:17) , t = ˜
O
(cid:16) 1 η (cid:17) and r =
˜ O (cid:16) log n η 2 (cid:17) .
Thus mtr = ˜
O (cid:16) √ n log n ( η − 11 .
6 ε ) η 3 (cid:17) and hence the total query complexity is ˜ O (cid:16) √ n log n ( η − 11 .
6 ε ) η 3 + n η 2 (cid:17) .
Evaluation
To evaluate the performance of Barbarik3 and test the quality of publicly available samplers, we implemented Barbarik3 in Python.
Our evaluation took inspiration from the experiments presented in previous work [14, 27], and we used the same framework to evaluate our proposed algorithm.
The role of target distribution P was played by WAPS 4 [23].
WAPS compiles the input Boolean formula into a representation that allows exact sampling and exact probability computation, thereby implementing the SAMP and EVAL oracles needed for our test.
For the role of sampler Q ( ϕ, w ) , we used the state-of-the-art samplers wSTS and wUnigen3 .
wUnigen3 [32] is a hashing-based sampler that provides ( ε, δ ) guarantees on the quality of the samples.
wSTS [20] is a sampler designed for sampling over challenging domains such as energy barriers and highly asymmetric spaces.
wSTS generates samples much faster than wUnigen3 , albeit without any guarantees on the quality of the samples.
To implement PCOND access, we use the
Kernel construction from [14].
Kernel takes in ϕ and two assignments σ 1 , σ 2 , and returns a function (cid:98) ϕ on m variables, such that: (1) m > n , (2) ϕ
and (cid:98) ϕ are similar in structure, and (3) for σ ∈ (cid:98) ϕ − 1 (1) , it holds that σ ↓ supp ( ϕ ) ∈ { σ 1 , σ 2 } .
Here σ ↓ supp ( ϕ ) denotes the projection of σ on the variables of ϕ .
For the closeness( ε ), farness ( η ) , and conﬁdence ( δ ) parameters, we choose the values 0 .
9 and 0 .
This setting implies that for a given distribution P , and for a given sampler Q ( ϕ, w ) , Barbarik returns (1) Accept if d ∞ ( P , Q ( ϕ, w )) < 0 .
05 , and (2) Reject if d TV ( P , Q ( ϕ, w )) > 0 .
9 , with probability at least 0 .
Our empirical evaluation sought to answer the question: How does the performance of Barbarik3 compare with the state-of-the-art tester Barbarik2 ?
Our experiments were conducted on a high-performance compute cluster with Intel Xeon(R)
E52690v3@2.60GHz CPU cores.
We use a single core with 4GB memory with a timeout of 16 hours for each benchmark.
We set a sample limit of 10 8 samples for our experiments due to our limited computational resources.
The complete experimental data along with the running time of instances, is presented in the Appendix B.
4.1 Setting A - scalable benchmarks
Dataset Our dataset consists of the union of two n -dimensional product distributions, for n ∈ { 4 , 7 , 10 , . .
We have 39 problems in the dataset.
We represent the union of two product distributions as the constraint: ϕ ( σ )
= (cid:86) 2 ki
=1 ( σ 3 k +1 ∨ σ i ) ∧ (cid:86) 3 ki =
k
+
( ¬ σ 3 k +1 ∨ σ i ) , and the weight function: w ( σ ) =
(cid:81) 3 ki =
k
+1 3 σ i , where σ i is the value of σ at position i .
Results We observe that in the case of wSTS , Barbarik2 can handle only 12 instances within the sample limit of 10 8 .
On the other hand, Barbarik3 can handle all 39 instances using at the most 10 6 samples.
In the case of wUnigen3 , Barbarik2 solves 5 instances, and Barbarik3 can handleinstances.
Figure 1 shows a cactus plot comparing the sample requirement of Barbarik3 and Barbarik2 .
The x -axis represents the number of benchmarks and y -axis represents the number of samples, a point ( x, y ) implies that the relevant tester took less than y number of samples to distinguish between d TV ( P , Q ( ϕ, w )) >
η and d ∞ ( P , Q ( ϕ, w ))
< ε , for x many benchmarks.
We display the set of benchmarks for which at least one of the two tools terminated within the sample limit of 10 8 .
We want to highlight that the y -axis is in log-scale, thus showing the sample efﬁciency of Barbarik3 compared to Barbarik2 .
For every benchmark, we compute the ratio of the number of samples required by
Barbarik2 to test a sampler and the number of samples required by Barbarik3 .
The geometric mean of these ratios indicates the mean speedup.
We ﬁnd that the Barbarik3 ’s speedup on wSTS is 451 × and on wUnigen3 is 10 × .
4.2 Setting B - real-life benchmarks
Dataset We experiment on 87 constraints drawn from a collection of publicly available benchmarks arising from sampling and counting tasks 5 .
We use distributions from the log-linear family.
In a loglinear distribution, the probability of an element σ ∈ ϕ − 1 (1) is given as: Pr[ σ ] ∝ exp ( (cid:80) ni
=
σ i θ i ) , where θ i ∈ R n> 0 .
We found that wUnigen3 was not able to sample from most of the benchmarks in the dataset within the given time limit, and hence we present the results only for wSTS .
Results
We ﬁnd that Barbarik3 terminated with a result on all 87 instances from the set of real-life benchmarks, while Barbarik2 could only terminate on 16 .
We present the results of our experiments in Table 1.
The ﬁrst column indicates the benchmark’s name, and the second column has the number of dimensions of the space the distribution is deﬁned on.
The third and ﬁfth columns indicate the number of samples required by Barbarik2 and Barbarik3 . The fourth and sixth columns report the output of Barbarik2 and Barbarik3 .
1010101010# o f s a m p l e s ( l og s ca l e ) wSTS
Barbarik
Barbarik          # of instances 10101010# o f s a m p l e s ( l og s ca l e ) wUnigen
Barbarik
Barbarik
Barbarik
Barbarik
[Table]
Dimensions
Result # of samples Result # of samples [Table] [Table]
R
R Pollard. [Table] R
R s
[Table]
[Table]
R
R s526a_3_
R
R s510_15_
R
R s27_new_7_
A
R s298_15_
R
R s420_3_
TOR s382_3_
TOR s641_3_
TOR 111.sk_2_
TOR 7.sk_4_
TOR 56.sk_6_
TOR s820a_15_
TOR
ProjectService3.sk_12_
TOR
Conclusion
In this paper, we studied the problem of testing constrained samplers over high-dimensional distributions with ( ε, η, δ ) guarantees.
For n -dimensional distributions, the existing state-of-the-art testing algorithm, Barbarik2 , has a worst-case query complexity that is exponential in n and hence is not ideal for use in practice.
We provided an exponentially faster algorithm, Barbarik3 , that has a query complexity linear in n and hence can easily scale to larger instances.
We implemented Barbarik and tested the samplers
wSTS and wUnigen
to determine their sample complexity in practice.
The results demonstrate that Barbarik3 is signiﬁcantly more sample efﬁcient than Barbarik2 , requiring 450 × fewer samples when it tested wSTS and 10 × fewer samples when it tested wUnigen3 .
Since there is a √ n gap between the upper bound provided by our work and the lower bound shown in [30], the problem of designing a more sample efﬁcient algorithm or ﬁnding a stronger lower bound, remains open.
Limitations For a given farness parameter η , Barbarik3 requires the value of the closeness parameter ε to lie in the interval [0 , η/ 11 . 6) .
In the case of Barbarik2 , the previous state-of-the-art test, the permissible values of ε for a given η lie in the interval [0 , η/ 3) .
Thus, Barbarik3 supports testing with only a subset of parameter values that Barbarik2 support.
Acknowledgments and Disclosure of Funding
We are grateful to the anonymous reviewers of NeurIPS 2022 for their constructive feedback that greatly improved the paper. We thank Ayush Jain and Shyam Narayanan for helpful discussions regarding the paper.
We would also like to thank Anna L.D.
Latour and Priyanka Golia for their useful comments on the code and the earlier drafts of the paper. This work was supported in part by National Research Foundation Singapore under its Campus for Research Excellence and Technological Enterprise (CREATE) programme, NRF Fellowship Programme [NRF-NRFFAI1-2019-0004] and Ministry of Education Singapore Tier 2 grant [MOE-T2EP201210011], Ministry of Education Singapore Tier 1 Grant [R-252-000-B59-114]. The computational work for this article was performed on resources of the National Supercomputing Centre, Singapore
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] (c) Did you discuss any potential negative societal impacts of your work?
[Yes] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] (b) Did you specify all the training details (e
g., data splits, hyperparameters, how they were chosen)?
[Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[N/A] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] (b) Did you mention the license of the assets?
[Yes] (c) Did you include any new assets either in the supplemental material or as a URL?
[Yes] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
A Missing proofs and algorithm A.1 Proof of Lemma
Proof.
The Hellinger distance of distributions P , Q restricted to a set S ⊆ { 0 , 1 } n , is deﬁned as
[Equation]
Then using the fact that d 2 H ( S ) ( P , Q ) ≤
d TV ( S ) ( P , Q )
we see that, (cid:80) i ∈ S (cid:112) P ( i ) Q
( i ) ≥
P ( S )+ Q ( S )− d TV ( S ) ( P , Q ) .
Then we use the Cauchy-Schwarz inequality:
[Equation]
A.2 Proof of Lemma
Lemma 2.
[Equation]
Proof.
Let PQ be a distribution constructed from P and Q , where we ﬁrst sample j ∼ B Q and then sample i ∼ P S j , thus PQ ( i ) =
(cid:80) j ∈ [ k ] ∪{ 0 }
B Q ( j ) P S j ( i ) .
We know that if i ∈ S j , then i (cid:54)∈ S j (cid:48) for j (cid:48) (cid:54) =
j .
This allows us to simplify and write PQ ( i ) =
B Q ( j ) P S
j
( i ) .
Then,
[Equation]
Since d TV ( B P , B Q ) < ε 2 , we have d TV ( P , PQ ) <
ε 2 .
From the deﬁnition of TV
[Equation]
we have
[Equation]
We will need the following sets: R 1 = { j : B P ( j ) >
[Equation]
B Q ( j ) } , R 2 = { j : B Q ( j
[Equation]
>
[Equation]
B P ( j ) }
[Equation]
From the triangle inequality we have d TV ( P
[Equation]
Q ) ≤
d TV ( P, PQ )
+ d TV ( PQ, Q ) .
We also know that d TV ( P, PQ ) <
ε 2 and d TV ( P , Q ) >
η .
Thus we have:
[Equation]
By deﬁnition, if j ∈ [ k ] \{ Bad ∪ R 1 ∪ R 2 } , then j has the property that d TV ( P S j , Q S j ) ≤ ε 1 .
Then, η − ε 2 < (cid:88)
[Equation]
Bad ) η − ε 2 − ε 1 − B Q ( { 0 } ∪ R 1 ∪ R 2 )
<
B Q ( Bad ) (2)
If i ∈ R 1 , then B P ( i ) > 2 B Q ( i ) , and thus B P ( i ) − B Q ( i ) > B Q ( i ) .
And thus,
B Q ( R 1 ) < (cid:88) i ∈ R( B P ( i ) − B Q ( i )) (3)
And if i ∈ R 2 , then B Q ( i ) > 5 B P ( i ) , and thus B Q ( i ) − B P ( i ) > 4 B P ( i ) , giving
[Equation]
B
[Equation]
P
[Equation]
i
[Equation]
Since | S 0 | ≤ 2 n and all elements i ∈ S 0 satisfy P ( i ) ≤ 2 − k , we have B P (0) ≤ 2 n − k , where we substitute k = n + log 2 (100 /η ) to get
[Equation]
Then,
[Equation]
Using (3),( 4)) and (5)
[Equation]
η
[Equation]
+ (cid:88)
[Equation]
Here we partition the set Bad ∪ { 0 } into two sets Bad + and Bad − , where Bad + = { i ∈
Bad ∪ { 0 }| B P ( i ) ≥ B Q ( i ) } and similarly Bad − = { i ∈
Bad ∪ { 0 }| B P ( i ) <
B Q ( i ) } .
B
[Equation]
[-R 
-]Q
[Equation]
(From 2) >
[Equation]
i ∈
[Equation]
(From 6) > 
[Equation] [Equation]

 And using the fact that d TV ( B P , B Q ) =
(cid:80) i : B P ( i ) ≤ B Q ( i ) ( B Q ( i ) − B P
( i ))
= (cid:80) i : B P ( i ) >
B Q ( i ) ( B P ( i ) − B Q ( i )) =
ε 2 , and the fact that ∀ i ∈ R 1 B P ( i ) >
B Q ( i ) and ∀ i ∈ R 2 B Q ( i ) >
B P ( i ) we have,
[Equation]
A.3 Proof of Lemma
Recall that for all r, s ∈ [ m ] , E [ X r,s ]
= (cid:80)
b ∈ Bad B P ( b ) B Q ( b ) .
Then since X = (cid:80) r,s ∈ [ m ] X r,s ,
E [ X ] = (cid:80) r,s ∈ [ m ] E [ X r,s ]
= m 2 E [ X r,s ] .
Then for i, j, k, l ∈ [ m ] ,
• if i = k, j = l then E [ X i,j X k,l ] = (cid:80) b ∈ Bad B P ( b ) B Q ( b ) =
E [ X r,s ] • if i = k, j (cid:54) =
l then E [ X i,j X k,l ] = (cid:80) b ∈ Bad B P ( b ) B 2 Q ( b ) • if i (cid:54) =
k, j = l then E [ X i,j X k,l ] = (cid:80)
b ∈ Bad B 2 P ( b ) B Q ( b ) • if i (cid:54) =
k, j (cid:54) =
l then E [ X i,j X k,l ] = (cid:0)(cid:80)
b ∈ Bad B P ( b ) B Q ( b ) (cid:1) 2 = E [ X r,s ]
E [ X 2 ] =
E   (cid:88) i,j,k,l ∈ [ m ]
X i,j X k,l   =
E   (cid:88) a (cid:54) =
c,b (cid:54) =
d i,j,k,l ∈ [ m ]
X i,j X k,l   + E   (cid:88) a = c,b (cid:54) =
d i,j,k,l ∈ [ m ]
X i,j X k,l   + E   (cid:88) a (cid:54) =
c,b = d i,j,k,l ∈ [ m ]
X i,j X k,l   + E   (cid:88)
a = c,b = d i,j,k,l ∈ [ m ]
X i,j X k,l   =
m 2 ( m − 1) 2 E
[ X r,s ] 2 + m 2 ( m − 1) (cid:32) (cid:88)
b ∈ Bad ( B P ( b ) + B Q ( b ))
B P ( i ) B Q ( i ) (cid:33) + m 2 E [ X r,s ] ≤ m 4 E [ X r,s ] 2 + m(cid:32) (cid:88)
b ∈ Bad ( B P ( b ) + B Q ( b ))
B P ( b ) B Q ( b ) (cid:33) + m 2 E [ X r,s ]
Then,
[Equation]
We will now focus on ﬁnding the maximum for ratio of summations:
[Equation]
(If b ∈ Bad then B P ( b ) /B Q ( b ) ∈ [5 − 1 , 2] )
[Equation]
(Using the monotonicity of (cid:96) p norms)
[Equation]
r,s ] − 1 /
[Equation]
Thus,
E
[Equation]
(Since m 2 E [ X r,s ] ≥ 1 )
The Chebyshev bound from 2 tells us that Pr[ | X − E
[ X ] | < E [ X ]] > E [ X ] 2 / E
[ X 2 ] > 15 .
Thus,
[Equation]
(Since X takes only integer values)
A.4 Proof of Proposition
Proposition 5.
Let P , Q be distributions and let p ∼ P and q ∼ Q .
Then,
If d ∞ ( P , Q ) < ε then
Q ( p ) Q ( p ) + Q ( q ) ≥ P ( p ) P ( p ) + (1 +
ε 1 − ε ) P ( q )
If d TV ( P , Q ) > ε 1 , then for 0 ≤ α < ε 1 , with probability at least ( d TV ( P , Q ) − α ) / 2 , Q ( p )
Q ( p
[Equation]
+ Q ( q
[Equation]
< P ( p ) [Equation] ( p
[Equation] 1 + [Equation] P ( q
[Equation]
Proof. If d ∞ ( P , Q ) < ε then Q ( p )
[Equation]
and hence we show the ﬁrst part of the claim.
For the second part of the proof we introduce the some sets.
Let H
= { h | 1 ≤ Q ( h ) P ( h ) <
1 + α } and H 1 = { h | 1 + α ≤ Q ( h ) P ( h ) } and H
= {+H 
+} ∪ H 1 .
Similarly deﬁne, L 0 = { (cid:96) | 1 − α < Q ( (cid:96) )
P ( (cid:96) ) < 1 } ,
L 1 = { (cid:96) | Q ( (cid:96) ) P ( (cid:96) ) ≤ 1 − α } and L = L 0 ∪ L 1 .
Now consider that we have a pair of samples, p ∼ P and q ∼ Q .
We know that either P ( L ) ≥ 1 /or P ( H ) > 1 / 2 .
P ( L ) ≥ 1 / 2 :
We see that Pr[ p ∈ L ] ≥ 1 / 2 .
Then from the deﬁnition of h 0 , Q ( h 0 ) − P ( h 0 ) < α and recall that Q ( H ) −P ( H ) =
d TV ( P , Q ) .
Thus we have that Q ( H 1 )
−P ( H 1 ) >
d TV ( P , Q ) − α and hence Pr[ q ∈ H 1 ] > d TV ( P , Q ) − α .
We can now conﬁrm that q ∈ H 1 ∧ p ∈ L with probability at least ( d TV ( P , Q ) − α ) / 2 .
Then,
Q ( p )
Q ( p ) + Q ( q ) <
P ( p )
P ( p ) + Q ( q ) (From P ( p ) > Q ( p ) ) <
P ( p )
P ( p ) + (1 + α ) P ( q ) (Since q ∈ H 1 )
P ( H ) > 1 / 2 : We see that Pr[ q ∈ H ] ≥ 1 / 2 .
Then we have that P ( L 0 )
−Q ( L 0 ) <
α and also that
P ( L ) − Q ( L ) =
d TV ( P , Q ) , we have that P ( L 1 ) − Q
( L 1 ) <
d TV ( P , Q ) − α .
Then, we deduce that probability at least ( d TV ( P , Q ) − α ) / 2 , q ∈ H ∧ p ∈
L 1 .
Then,
Q ( p )
Q ( p ) + Q ( q ) <
Q ( p )
Q ( p ) + P ( q ) (From P ( q ) < Q ( q ) ) <
P ( p )(1 − α )
P ( p )(1 − α ) + P ( q ) (Since p ∈ L 1 ) <
P ( p )
P ( p ) + (1 + α ) P ( q )
A.5 The outbucket subroutine
1: Sample max (cid:16) 4( k +1) θ, 8ln(4 /δ ) θ(cid:17) times from B P and B Q and construct empirical distributions (cid:99) B P and (cid:100) B Q .
2: Return d TV ( (cid:99) B P , (cid:100) B Q )
B Data missing from the main paper
[Table]
TO0.
R
[Table]
[Table]
sk_3_
TO2.
R  s344_3_
TO0.
R  s526_3_
TO0.
R
[Table]
53.sk_4_
[Table]
R  s420_7_
[Table]
R  10.sk_1_
[Table]
R
[Table]
sk_3_
[Table]
[Table] R  s349_15_ [Table] R  s820a_7_
[Table]
R  s832a_15_
[Table]
R
[Table]
sk_2_ [Table] R  s344_15_ [Table]
R [Table] sk_5_
TO6. R  s420_3_ [Table]
R  s420_new1_15_
TO0. R
UserServiceImpl.sk_8_
TO0.
R
[Table] sk_2_ TO0. R  s349_3_
TO0.
R  s953a_7_
TO0.
R  s444_7_
TO1.
R  77.sk_3_
TO0.
R  51.sk_4_
TO1.
R  109.sk_4_
TO0.
R  s832a_7_
TO0. R  s526a_15_ TO86. R  s420_new_
TO0. R  s382_3_ TO0. R  s641_3_
TO19.
R  s420_new_3_
TO0.
R
LoginService2.sk_23_
TO18.
R  s832a_3_
TO0.
R  s420_new_15_
TO0.
R  s420_new1_3_
TO0.
R  s838_3_
TO0.
R  70.sk_3_
TO0.
R  s820a_15_
TO0.
R
29.sk_3_
TO3.
R  19.sk_3_
TO3.
R  57.sk_4_
TO0.
R  s444_15_
TO0.
R  s1238a_3_
TO0.
R  s526_7_
TO46.
R  s382_7_ TO0. R  s1238a_7_ TO1.
R  7.sk_4_ TO0.
R  55.sk_3_
TO0.
R  s713_7_
TOR  s420_new1_7_
TO0.
R  s641_7_
TOR  s1196a_15_
TO2.
R
ProjectService3.sk_12_
TOR  s1196a_3_
TO0.
R  s1238a_15_
TO2.
R  s526_15_
TOR  s820a_3_ TO0.
R  27.sk_3_ TO0. R  s510_3_
TO0. R  s1196a_7_
TO1. R  s344_7_ TO0.
R  s713_3_ TO44. R  s953a_3_ TO0.
R  s526a_7_
TO44. R  s420_15_
TO0.
R  s953a_15_ TO0.
R  s838_7_ TO0. R  56.sk_6_
TO0. R  32.sk_4_
TO0.
R  s382_15_ TO8. R  s1488_3_
TO0. R [Table] sk
[Table] [Table] TO3. R
In the table ‘TO’ represents that either the tester timed out or asked for more than 10 8 samples. The closeness = 05 , for farness is η = 0 .
9 and for conﬁdence is δ = 0 .