A Scalable Tester for Samplers
Anonymous Author(s)
AfﬁliationAddress email
Abstract
In this paper we study the problem of testing of constrained samplers over highdimensional distributions with ( ε, η, δ ) guarantees.
Samplers are increasingly usedin a wide range of safety-critical ML applications, and hence the testing problemhas gained importance.
For n -dimensional distributions, the existing state-of-the-artalgorithm, Barbarik2 , has a worst case query complexity of exponential in n andhence is not ideal for use in practice.
Our primary contribution is an exponentiallyfaster algorithm that has a query complexity linear in n and hence can easily scaleto larger instances.
We demonstrate our claim by implementing our algorithm andthen comparing it against Barbarik2 .
Our experiments on the samplers wUnigen3and wSTS , ﬁnd that Pacoco requires 10 × fewer samples for wUnigen3 and 450 ×fewer samples for wSTS as compared to Barbarik2 .
Introduction
The constrained sampling problem is to draw samples from high-dimensional distributions overconstrained spaces.
A constrained sampler Q ( ϕ, w ) takes in a set of constraints ϕ : { 0 , 1 } n → { 0 , 1 }and a weight function w : { 0 , 1 } n → R > 0 , and returns a sample σ ∈ ϕ − 1 (1) with probability proportional to w ( σ ) .
Constrained sampling is a core primitive of many statistical inference methods usedin ML, such as Sequential Monte Carlo[29], Markov Chain Monte Carlo(MCMC)[3, 9], Simulated
Annealing [4], and Variational Inference [24].
Sampling from real-world distributions is often computationally intractable, and hence, in practice, samplers are heuristical and lack theoretical guarantees.
For such samplers, it is an important problem to determine whether the sampled distribution is closeto the desired distribution, and this problem is known as testing of samplers .
The problem wasformalised in [14, 27] as follows: Given access to a target distribution P , a sampler Q ( ϕ, w ) , andthree parameters ( ε, η, δ ) , with probability at least 1 − δ , return (1) Accept if d ∞ ( P , Q ( ϕ, w )) < ε ,or (2) Reject if d TV ( P , Q ( ϕ, w )) >
η .
Here d TV is the total variation distance, d ∞ the multiplicativedistance, and ε, η, and δ are parameters for closeness, farness and conﬁdence respectively.
Accessto distribution P is via the DUAL oracle, and access to Q is via the PCOND and SAMP oracles(deﬁned in Section 2.1)
There is substantial interest in the testing problem due to the increasing use of ML systems inreal-world applications where safety is essential, such as medicine [2], transportation [8, 25], andwarfare [28].
For the ML systems that incorporate samplers, the typical testing approach has beento show the convergence of the sampler with the target distribution via empirical tests that rely onheuristics and do not provide any guarantees [19, 22, 31, 34].
In a recent work [27], a novel framework,called Barbarik2 , was proposed that could test a given sampler while providing ( ε, η, δ ) guarantees,using ˜ O (cid:16) tilt ( P ) 2 η ( η − 3 ε ) 3 (cid:17) queries, where tilt ( P ) :=
[Equation]
P (
[Equation]
1 ) P ( σ 2 ) for P ( σ 2 ) > 0 .
Since the tilt ( P ) can take arbitrary values, we observe that the query complexity can be prohibitively large 1 .
On theother hand, the best known lower bound for the problem, derived from [30], is ˜Ω (cid:18) √ n/ log( n ) η(cid:19) .
Inthis work, we take a step towards bridging this gap with our algorithm, Pacoco , that has a querycomplexity of ˜ O (cid:16) √ n log n ( η − 11 .
6 ε ) η 3 + nη 2 (cid:17) , representing an exponential improvement over the state of theart.
To be of any real value, testing tools must be able to scale to larger instances.
In the case of constrainedsamplers, the only existing testing tool, Barbarik2 , is not scalable owing to its query complexity.
Thelack of scalability is illustrated by the following fact: product distributions are the simplest possibleconstrained distributions, and given a union of two n -dimensional product distributions, Barbarik2requires more than 10 8 queries for n > 30 .
On the other hand, the query complexity of Pacoco scaleslinearly with n , the number of dimensions, thus making it more appropriate for practical use.
We implement Pacoco and compare it against Barbarik2 to determine their relative performance.
In our experiments, we consider two sets of problems, (1) constrained sampling benchmarks, (2)scalable benchmarks and two constrained samplers
wSTS and wUnigen3 .
We found that to completethe test Pacoco required at least 450 × fewer samples from wSTS and 10 × fewer samples fromwUnigen3 as compared to Barbarik2 .
Moreover, Pacoco terminates with a result on at least 3 × morebenchmarks than Barbarik2 in each experiment.
Our contributions can be summarized as follows:
For the problem of testing of samplers, we provide an exponential improvement in querycomplexity over the current state of the art test Barbarik2 .
Our test, Pacoco , makes a totalof ˜ O (cid:16) √ n log n ( η − 11 .
6 ε ) η 3 + nη 2 (cid:17) queries, where ˜ O hides polylog factors of ε, η and δ .2.
We present an extensive empirical evaluation of Pacoco and contrast it with Barbarik2 .
Theresults indicate that Pacoco requires far fewer samples and terminates on more benchmarkswhen compared to Barbarik2 .
We deﬁne the notation and discuss related work in Section 2.
We then present the main contributionof the paper, the test Pacoco , and its proof of correctness in Section 3.
We present our experimentalﬁndings in Section 4 and then we conclude the paper and discuss some open problems in Section 5.
Due to space constraints, we defer some proofs and the full experimental results to the supplementary
Section A and B respectively.
Notation and preliminaries
Probability distributions In this paper we deal with samplers that sample from discrete probabilitydistributions over high-dimensional spaces.
We consider the sample space to be the n -dimensional
Boolean hypercube { 0 , 1 } n .
A constrained sampler Q takes in a set of constraints ϕ : { 0 , 1 } n →{ 0 , 1 } and a weight function w : { 0 , 1 } n → R > 0 , and samples from the distribution Q ( ϕ, w ) deﬁned as Q ( ϕ, w )( σ ) = (cid:26) w ( σ ) / w ( ϕ ) σ [Equation] ϕ − 1 (1)
[Equation]
0) , where w ( ϕ ) = (cid:80)
[Equation]
ϕ
[Equation] [Equation]
1) w ( σ ) .
To improve readability,we use Q to refer to the distribution Q ( ϕ, w ) .
For an element i , D ( i ) denotes it’s probability indistribution D
and i ∼ D represents that i is sampled from D .
For any non-empty set S ⊆ { 0 , 1 } n ,
D S is the distribution D conditioned on set S , and D ( S ) is the probability of S in D i.e., D ( S ) =
72 (cid:80) i ∈ S D ( i ) .
The total variation (TV) distance of two probability distributions D 1 and D 2 is deﬁned as:d TV ( D [Equation] , D [Equation]
) = [Equation] i ∈{ 0 , 1 } n |D 1 ( i ) − D 2 ( i ) | .
For S ⊆ { 0 , 1 } n , we deﬁne d TV ( S )
( D 1 , D 2 )
=12 (cid:80) i ∈ S |D 1 ( i ) − D 2 ( i ) | .
The multiplicative distance of D 2 from D 1 is deﬁned as: d ∞ ( D 1 , D 2 ) =
max i ∈{ 0 , 1 } n |D 2 ( i ) /
D 1 ( i ) − 1 | .
The two notions of distance obey the identity: 2 d TV ( D 1 , D 2 ) ≤d ∞ ( D 1 , D 2 ) .
In the rest of the paper, E [ v ] represents the expectation of random variable v and [ k ] represents theset { 1 , 2 . . .
, k } .
Tools used in the analysis
Proposition 1 (Hoeffding) .
For independent 0-1 random variables X i , X
= (cid:80) ki
=1 X i , and t ≥ 0 ,
Pr( X − E X > t ) ≤ exp (cid:0) − 2 t 2 k (cid:1) and Pr( E X − X > t ) ≤ exp (cid:0) − 2 t 2 k (cid:1)
Proposition 2 (Chebyshev) .
Given bounded r.v. X , we have Pr( | X − E
[ X ] | < E [ X ]) >
E [ X ] 2 E [ X 2 ]
Proposition 3.
Given distributions D 1 and D 2 supported on { 0 , 1 } n , and a set S ⊆ { 0 , 1 } n ,(cid:88)
i
[Equation]
The proof can be found in the Appendix A.
If we are given samples { s 1 , s 2 , . . .
, s n } from a distribution D over [ k ] , then the empirical distribution(cid:98)
D is deﬁned to be (cid:98) D ( i ) =
1 n k (cid:80) j = 1 { s j = i } .
Proposition 4 (See [11] for a simple proof) .
Suppose D is a distribution over [ k ] , and (cid:98) D is constructedusing max (cid:16) kη 2 ,
2ln(2 /δ ) η(cid:17) samples from D .
Then d TV ( D , (cid:98) D ) ≤
η with probability at least 1 − δ
2.1 Testing with the help of oracles
In distribution testing, we are given samples from an unknown distribution P over a large support{ 0 , 1 } n , and the task is to test whether P satisﬁes some property of interest.
One of the importantproperties we care about is whether P is close to another distribution Q , and this subﬁeld of testing isknown as closeness testing .
It was shown by Valiant and Batu et al.
that the ability to draw samplesfrom P and Q is not powerful enough, as at least Ω(2 2 n/ 3 ) samples are required to provide any sortof probabilistic guarantee for closeness testing.
Since n is usually large, it was desirable to ﬁnd teststhat could solve the closeness testing problem using polynomially many samples in n .
Motivated by the above requirement, Canonne et al.
and Chakraborty et al. introduced the conditionalsampling oracle ( COND ), that is a more powerful way to access distributions.
A COND oracle fordistribution D over { 0 , 1 } n takes as input a set S ⊆ { 0 , 1 } n with D ( S ) > 0 , and returns a samplei ∈ S with probability D ( i ) / D ( S ) .
It has been shown that the use of the COND oracle, and itsvariants, drastically reduces the sample complexity of many tasks in distribution testing [1, 21, 12, 15,6, 26, 7, 17, 13, 30] (see [10] for an extensive survey).
In this paper, we consider the pair-conditioning( PCOND ) oracle, which is a special case of the COND oracle with the restriction that | S | = 2 i.e.,the size of the conditioning set has to be two.
To engineer practical PCOND oracle access intoconstrained samplers, we use the chain formula construction introduced in [14].
With the same goal of designing tests with polynomial sample complexity, a different kind of oracle,known as the DUAL oracle, was proposed by Canonne et al..
The DUAL oracle allows one to samplefrom a given distribution and also query the distribution for the probability of arbitrary elements of thesupport.
Tractable DUAL oracle access is supported by a number of distribution representations, suchas the fragments of probabilistic circuits (PC) that support the EVI query [18].
In our experimentalevaluation, we use distributions from one such fragment: weighted d - DNNF s. Weighted d - DNNF sare a class of arithmetic circuits with properties that enable DUAL oracle access in time linear in thesize of the circuit [16, 23].
Pacoco : an algorithm for testing samplers
We start by providing a brief overview of our testing algorithm before providing the full analysis Algorithm
1: k ← n + (cid:100) log 2 (100 /η ) (cid:101) 2: for i = 1 to k do 3: S i = { b : 2 − i < P ( b ) ≤ 2 − i +1 } 4: S 0 = { 0 , 1 } n \ (cid:83) i ∈ [ k ] S i 5: B P is the distribution over [ k ] ∪ { 0 } where we sample i ∼ B P if we sample j ∼ P and j ∈ S i 6: B Q is the distribution over [ k ] ∪ { 0 } where we sample i ∼ B Q if we sample j ∼ Q and j ∈ S i 7: θ ← η/8: (cid:98) d ← OutBucket ( B P , B Q , k, θ, δ/ 2) 9: if (cid:98) d > ε/ 2 + θ then 10: Return Reject 11: ε 2 ← (cid:98) d
+ θ 12: Return InBucket ( P , Q , k, ε, ε 2 , η, δ/ 2)
The pseudocode of Pacoco is given in Algorithm 1.
We adapt the deﬁnition of bucketing of distributions from [30] for use in our analysis.
Deﬁnition 1.
For a given k ∈ N > 0 , the bucketing of { 0 , 1 } n with respect to P is deﬁned as follows:
For 1 ≤ i ≤ k , let S i
= { b : 2 − i < P ( b ) ≤ 2 − i +1 } and let S
= { 0 , 1 } n \ (cid:83) i ∈
[ k ] S i .
Givenany distribution D over { 0 , 1 } n , we deﬁne a distribution B D over [ k ] ∪ { 0 } as: for 0 ≤ i ≤ k ,
B D ( i ) =
D ( S i ) .
We call B D the bucket distribution of D and S i the i th bucket.
Pacoco takes as input two distributions P and Q deﬁned over the support { 0 , 1 } n , along with theparameters for closeness( ε ), farness( η ), and conﬁdence( δ ).
On Line 1, Pacoco computes the value ofk using η and the number of dimensions n .
Then, using DUAL access to P , and SAMP access to Q ,
Pacoco creates bucket distributions B P and B Q as in Defn.
1, in the following way: To sample from
B P , Pacoco ﬁrst draws a sample j ∼ P , then using the DUAL oracle, determines the value of P ( j ) .
Then, if j lies in the
i th bucket i.e., 2 −
i < P ( j ) ≤ 2 − i +1 , the algorithm takes sample i as the samplefrom B P .
Similarly, to draw a sample from B Q , Pacoco draws a sample j ∼ Q and then, using the
DUAL oracle to ﬁnd P ( j ) , ﬁnds i such that j lies in the i th bucket, and then uses i as the sample.
Pacoco then calls two subroutines, OutBucket (Section 3.4) and InBucket (Section 3.3).
The
OutBucket subroutine returns an θ -multiplicative estimate of the TV distance between B P and
B Q , the two bucket distributions of P and Q , with an error of at most δ/ 2 .
If it is found on Line 9 thatthe estimate (cid:98) d is greater than ε/ 2 + θ , we know that d TV ( P , Q ) > ε/ 2 and also that d ∞ ( P , Q ) > ε ,and hence the algorithm returns Reject .
Otherwise, the algorithm calls the InBucket subroutine.
Now suppose that d TV ( P , Q ) ≥ η .
Then, for ε 2 (Line 11), it is either the case that d TV ( B P , B Q ) >
ε 2 or else d TV ( B P , B Q ) ≤ ε 2 .
In the former case, the algorithm returns Reject on Line 10, and inthe latter case the InBucket subroutine returns Reject .
In both cases, the failure probability is at mostδ/ 2 .
Thus Pacoco returns Reject on given η -far input distributions with probability at least 1 − δ .
We will now prove the following theorem:
Theorem 1.
Pacoco ( P , Q , η, ε, δ ) takes in distributions P and Q deﬁned over { 0 , 1 } n , and parameters η ∈ (0 , 1] , ε ∈ [0 , η/ 11 . 6) and δ ∈ (0 , 1 / 2] . Pacoco has DUAL access to P , and PCOND + SAMP access to Q .
With probability at least 1 − δ , Pacoco returns
• Accept if d ∞ ( P , Q ) ≤ ε• Reject if d TV ( P , Q ) >
η
Pacoco has query complexity ˜ O (cid:16) √ n log( n ) η 3 ( η − 11 .
6 ε ) + nη 2 (cid:17) , where ˜ O hides polylog factors of ε, η and δ . 3.2 Lower bound
The lower bound comes from the paper of Narayanan [30], where it appears in Theorem 1.6.
Phrasedin the jargon of our paper, the lower bound states that distinguishing between d TV ( P , Q ) > η andd ∞ ( P , Q ) = 0 requires ˜Ω( (cid:112) n/ log( n ) /η 2 ) samples.
Note that the lower bound is shown on a specialcase ( ε = 0 ) of our problem. Hence the lower bound applies to our problem as well. Furthermore, thelower bound is shown for the case where distribution P provides full access, i.e., the algorithm canmake arbitrary queries to P . This is a stronger access model than DUAL .
The lower bound is for astronger access model, hence it extends to our problem as well.
In this section, we present the InBucket subroutine, whose behavior is stated in the following lemma.
Lemma 1.
InBucket ( P , Q , k, ε, ε 2 , η, δ ) takes as input two distributions P , Q , an integer k andparameters ε, ε 2 , η, δ .
If d ∞ ( P , Q ) ≤ ε , InBucket returns Accept .
If d TV ( P , Q ) ≥ η andd TV ( B P , B Q ) <
ε 2 , then InBucket returns Reject .
InBucket errs with probability at most δ .
1: ε 1 ← (0 . 99 η − 3 .
25 ε 2 − 2 ε/ (1 − ε )) / 1 .
+ 2 ε/ (1 − ε ) [Table] m ← (cid:100)√ k/ (0 . 99 η − 3 .
25 ε 2 − ε 1 ) (cid:101) 3: α ← ( ε 1 + 2 ε/ (1 − ε )) / [Table] t ← (cid:108) ln [Table]
[Table] ) ln(10 / (10 − ε 1 + α )) (cid:109) [Table]
for
[Table] do [Table] 7: ∀ i ∈
[ k ] Γ i P
← Γ P ∩ S i (cid:46) S
i is deﬁned in Defn.8: Γ Q ← m samples from Q 9: ∀ i ∈ [ k ] Γ i Q ← Γ Q ∩ S
i 10: for all j ∈ [ k ] s.t.
| Γ j P | , | Γ j Q | > 0 do 11: p ← Γ j P (cid:46) p is an arbitrary sample from the set Γ j P 12: q ← Γ j Q (cid:46) q is an arbitrary sample from the set Γ j Q 13: h ←
P ( p ) P ( p )+ P ( q )(1+ 2 ε 1 − ε ) 14: (cid:96) ←
P ( p ) P ( p )+ P ( q )(1+ α ) 15: r ← (cid:108) 2ln(4 mt/δ ) ( h − (cid:96) ) 2 (cid:109) 16: (cid:98) c
← Bias ( Q, p, q, r ) 17: if (cid:98) c ≤ ( h + (cid:96) ) /
2 then 18:
Return Reject 19: Return Accept Algorithm
1: if p and q are identical then 2:
Return 0 .3:
Γ Q { p,q } ← r samples from Q { p,q } 4: Return # of times p appears in Γ Q { p,q }
InBucket makes extensive use of the PCOND oracle access to Q via the Bias subroutine, which wedescribe in the following subsection.
The Bias subroutine The Bias subroutine takes in distribution Q , two elements p, q and a positiveinteger r .
Then, using the PCOND oracle, Bias draws r samples from the conditional distribution
Q { p,q } and returns the number of times it sees p in the r samples. It can be seen that the returned value is an empirical estimate of
Q ( p ) Q ( p )+ Q ( q ) .
Let the estimate be (cid:99) c pq .
We use the Hoeffding bound in
Prop. 1, and the value of r from Line 15 of Alg.
(2) to show that:
[Equation]
Here t represents the number of iterations of the outer loop (Line 4), and m is the number of samplesdrawn from B P and B Q .
Together, there are at most mt pairs of samples that are passed to the
Bias oracle.
Since in each invocation of Bias , the probability of error is δ/ 4 mt , using the unionbound we ﬁnd that the probability that all mt Bias calls return correctly is at least 1 − δ/ 4 and thuswith probability at least 1 − δ/ 4 , the empirical estimate (cid:99) c pq is closer than ( h − (cid:96) ) / 4 to
Q ( p ) Q ( p )+ Q ( q ) .
Henceforth we assume:
[Equation]
3.3.1 The Accept case
In this section we will provide an analysis of the case when d ∞ ( P , Q ) < ε .
We will now state aproposition required for the remaining proofs, the proof of which we relegate to Appendix A.4.
Proposition 5.
Let P , Q be distributions and let p ∼ P and q ∼ Q .
Then,1.
If d ∞ ( P , Q ) < ε then Q
[Equation]
If d TV ( P , Q ) > ε 1 , then for 0 ≤ α < ε 1 , with probability at least ( d TV ( P , Q ) − α ) / 2 ,
[Equation]
From our assumption (1), we know that for all invocations of Bias , with probability at least 1 − δ/ 4 ,(cid:12)(cid:12)(cid:12)(cid:99) c pq −
Q ( p ) Q ( p )+ Q ( q ) (cid:12)(cid:12)(cid:12) ≤ ( h − (cid:96) ) / 2 .
Using Prop. 5, and using the value of h given on Line 13, we cansee that
Q ( p ) Q ( p )+ Q ( q ) > h .
From this we can observe that for all invocations of Bias , (cid:99) c pq > ( h + (cid:96) ) /
2and the test does not return Reject in any iteration, hence eventually returning Accept .
Thus, in thecase that d ∞ ( P , Q ) < ε , the InBucket subroutine returns Accept with probability at least 1 − δ/ 4 .
3.3.2 The Reject case
In this section we analyse the case when d TV ( P , Q ) ≥ η and d TV ( B P , B Q ) ≤ ε 2 and we will showthat the algorithm returns Reject with probability at least 1 − δ .
For the purpose of the proof we willdeﬁne a set of bad buckets Bad ⊆
[ k ] .
Note that bucket { 0 } is not in Bad .
Deﬁnition 2.
Bad =
{ i ∈ [ k ] : d TV ( P S i , Q S i ) >
ε 1 ∧ B P ( i ) /B
Q ( i ) ∈
[5 − 1 , 2] }
Suppose we have an indicator variable X r,s constructed as follows: draw m samples from P and Q ,and if the r th sample from P and the s th sample from Q both belong to some bucket b ∈ Bad , then
X r,s = 1 else X r,s = 0 .
Then,
[Equation]
The inequality is by the application of Prop. 3.
We analyse the expression for the expectation in the following lemma, the proof of which we relegateto Appendix A.
Lemma 2.
[Equation]
Using Lemma 2 we immediately derive the fact that E [ X r,s ] > (cid:0) 0 .
99 η − 134 ε 2 − ε 1 (cid:1) 2 /K .
Let
X = (cid:80) r,s ∈ [ m ] X r,s .
Given m samples from P and Q , Pr( X ≥ 1) is the probability that there is atleast one bucket in Bad that is sampled at least once each in both sets of samples.
Lemma 3.
Pr( X ≥ 1) > 1 /
The proof can be found in Appendix A.3.
Henceforth we will condition on the the event that X ≥ 1 .
In such a case, we know that for somek ∈ Bad , there is a sample p ∼ P S k and a sample q ∼ Q S k .
Then for such a pair of samples ( p, q ) ,and some α , Prop. 5 tells us that with probability at least ( d TV ( P , Q ) − α ) /
2 we have
[Equation]
Using the assumption made in (1), we immediately have that (cid:99) c pq ≤
Q ( p ) Q ( p )+ Q ( q ) + h − (cid:96) 2 .
But from
Prop.
we have that
Q ( p ) Q ( p )+ Q ( q ) < (cid:96) and hence (cid:99) c pq < ( h + (cid:96) ) / 2 .
Since d TV ( P , Q ) ≥ ε 1 , we see thatif X ≥ 1 , then with probability at least ( ε 1 − α ) / 2 , the iteration returns Reject .
Then, using Lemma 3 we see that in every iteration, with probability at least ( ε 1 − α ) / 10 , InBucketreturns Reject .
There are t iterations, where t (line 4) is chosen such that the overall probability ofthe test returning Reject is at least 1 − δ/ 2 .
3.4 The OutBucket subroutine
The OutBucket subroutine takes as input two distributions D 1 , D 2 over k + 1 elements and twoparameters θ and δ .
Then with probability at least 1 − δ , InBucket returns a θ -multiplicative estimatefor d TV ( D 1 , D 2 ) .
The OutBucket starts by drawing max (cid:16) 4( k +1) θ, 8ln(4 /δ ) θ(cid:17) samples from the two distributions D 1and D 2 , and constructs the empirical distributions (cid:99) D 1 and (cid:99) D 2 .
Then from Prop. 4, we know that withprobability at least 1 − δ , both d TV ( D 1 , (cid:99) D 1 ) ≤ θ/ 2 and d TV ( D 2 , (cid:99) D 2 ) ≤ θ/ 2 .
From the triangle inequality we have that,
[Equation]
and also that,
[Equation]
Thus with probability at least 1 − δ , the returned estimate d TV ( (cid:99) D 1 , (cid:99) D 2 ) satisﬁes | d TV ( (cid:99) D 1 , (cid:99) D 2 )
−d TV
( D 1 , D 2 ) | < θ .
Query and runtime complexity The number of queries made by OutBucket to P and Q isgiven by ˜ O (cid:16) nη 2 (cid:17) , where ˜ O hides polylog factors of ε, η and δ .
The number of queries requiredby InBucket is given by mtr .
Bounding the terms individually, we see that m =
˜ O (cid:16) √ n η − 11 .
6 ε (cid:17) ,t = ˜
O
(cid:16) 1 η (cid:17) and r =
˜ O (cid:16) log n η 2 (cid:17) .
Thus mtr = ˜
O (cid:16) √ n log n ( η − 11 .
6 ε ) η 3 (cid:17) and hence the total query complexityis ˜ O (cid:16) √ n log n ( η − 11 .
6 ε ) η 3 + nη 2 (cid:17) .
Evaluation
To evaluate the performance of Pacoco and test the quality of publicly available samplers, we implemented Pacoco in Python.
Our evaluation took inspiration from the experiments presented inprevious work [14, 27], and we used the same framework to evaluate our proposed algorithm.
The role of target distribution P was played by the exact constrained sampler WAPS 2 [23].
For the roleof sampler Q ( ϕ, w ) , we used the state-of-the-art samplers wSTS and wUnigen3 .
wUnigen3 [32] is ahashing-based sampler that provides ( ε, δ ) guarantees on the quality of the samples.
wSTS [20] is asampler designed for sampling over challenging domains such as energy barriers and highly asymmetric spaces.
wSTS generates samples much faster than wUnigen3 , albeit without any guaranteeson the quality of the samples.
For the closeness( ε ), farness ( η ) , and conﬁdence ( δ ) parameters, we choose the values 0 .
9 and 0 . 2 .
This setting implies that for a given distribution P , and for a given sampler Q ( ϕ, w ) , Pacoco returns(1) Accept if d ∞ ( P , Q ( ϕ, w )) < 0 .
05 , and (2) Reject if d TV ( P , Q ( ϕ, w )) > 0 .
9 , with probabilityat least 0 .
Our empirical evaluation sought to answer the question: How does the performance of
Pacoco compare with the state-of-the-art tester Barbarik2 ?
Our experiments were conducted on a high-performance compute cluster with Intel Xeon(R)
E52690v3@2.60GHz CPU cores.
We use a single core with 4GB memory with a timeout of 16 hoursfor each benchmark.
We set a sample limit of 10 8 samples for our experiments due to our limitedcomputational resources.
4.1 Setting A - scalable benchmarks
Dataset Our dataset consists of the union of two n -dimensional product distributions, for n ∈{ 4 , 7 , 10 , . .
We have 39 problems in the dataset.
We represent the union of two productdistributions as the constraint: ϕ ( σ )
= (cid:86) 2 ki
=1 ( σ 3 k +1 ∨ σ i ) ∧ (cid:86) 3 ki =
k
+
( ¬ σ 3 k +1 ∨ σ i ) , and the weightfunction: w ( σ ) =
(cid:81) 3 ki =
k
+1 3 σ i , where σ i is the value of σ at position i .
Results We observe that in the case of wSTS , Barbarik2 can handle only 12 instances within thesample limit of 10 8 .
On the other hand, Pacoco can handle all 39 instances using at the most 10 6samples.
In the case of wUnigen3 , Barbarik2 solves 5 instances, and Pacoco can handle 17 instances.
Figure 1 shows a cactus plot comparing the sample requirement of Pacoco and Barbarik2 .
Thex -axis represents the number of benchmarks and y -axis represents the number of samples, a point( x, y ) implies that the relevant tester took less than y number of samples to distinguish betweend TV ( P , Q ( ϕ, w ))
>
η and d ∞ ( P , Q ( ϕ, w ))
< ε , for x many benchmarks.
We display the set ofbenchmarks for which at least one of the two tools terminated within the sample limit of 10 8 .
We wantto highlight that the y -axis is in log-scale, thus showing the sample efﬁciency of Pacoco comparedto Barbarik2 .
For every benchmark, we compute the ratio of the number of samples required by
Barbarik2 to test a sampler and the number of samples required by Pacoco .
The geometric mean ofthese ratios indicates the mean speedup.
We ﬁnd that the Pacoco ’s speedup on wSTS is 451 × and onwUnigen3 is 10 × . 4.2 Setting B - real-life benchmarks
{+Dataset We experiment on 87 constraints drawn from a collection of publicly available benchmarksarising from sampling and counting tasks 3 . We use distributions from the log-linear family. In a loglinear distribution, the probability of an element σ ∈ ϕ − 1 (1) is given as: Pr[ σ ] ∝ exp ( (cid:80) ni =
+}
σ i θ i ) ,where θ i ∈ R n> 0 . We found that wUnigen3 was not able to sample from most of the benchmarks inthe dataset within the given time limit, and hence we present the results only for wSTS . Results We ﬁnd that Pacoco terminated with a result on all 87 instances from the set of real-lifebenchmarks, while Barbarik2 could only terminate on 16 .
We present the results of our experimentsin Table 1. The ﬁrst column indicates the benchmark’s name, and the second column has the numberof dimensions of the space the distribution is deﬁned on. The third and ﬁfth columns indicate the 1010101010# o f s a m p l e s ( l og s ca l e ) wSTS
Barbarik
Pacoco         # of instances 10101010# o f s a m p l e s ( l og s ca l e ) wUnigen
Barbarik
Pacoco number of samples required by Barbarik2 and Pacoco . The fourth and sixth columns report the outputof Barbarik2 and Pacoco .  Conclusion
In this paper, we studied the problem of testing constrained samplers over high-dimensional distributions with ( ε, η, δ ) guarantees. For n -dimensional distributions, the existing state-of-the-art testingalgorithm, Barbarik2 , has a worst-case query complexity that is exponential in n and hence is notideal for use in practice.
We provided an exponentially faster algorithm, Pacoco , that has a querycomplexity linear in n and hence can easily scale to larger instances.
[-=
 σ i θ i ) ,where θ i ∈ R n> 0 .-] We implemented Pacoco andtested the samplers
{+wSTS and wUnigen
+}
to determine their sample complexity in practice. The resultsdemonstrate that Pacoco is signiﬁcantly more sample efﬁcient than Barbarik2 , requiring 450 × fewersamples when it tested wSTS and 10 × fewer samples when it tested wUnigen3 .
Since there is a √ ngap between the upper bound provided by our work and the lower bound shown in [30], the problemof designing a more sample efﬁcient algorithm or ﬁnding a stronger lower bound, remains open.
Limitations For a given farness parameter η , Pacoco requires the value of the closeness parameterε to lie in the interval [0 , η/ 11 . 6) . In the case of Barbarik2 , the previous state-of-the-art test, thepermissible values of ε for a given η lie in the interval [0 , η/ 3) .
Thus, Pacoco supports testing withonly a subset of parameter values that Barbarik2 supports.
[Table]
Checklist
For all authors...
(a) Do the main claims
made in the abstract and introduction accurately reﬂect the paper’s
contributions and[-wUnigen
-]
scope? [Yes] (b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work? [Yes] (d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes] If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e g., data splits, hyperparameters, how they were chosen)?
[Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[N/A]
(d) Did you include the total amount of compute and the type of resources used (e
g., type
of GPUs, internal cluster, or cloud provider)?
[Yes]
If you are using existing assets (e g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[Yes]
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation? [N/A] A Missing proofs and algorithmA.1 Proof of Lemma 3 Proof.
The Hellinger distance of distributions P , Q restricted to a set S ⊆ { 0 , 1 } n , is deﬁned as
[Equation] [Equation] i [Equation]
Then using the fact that d 2 H ( S ) ( P , Q ) ≤
d TV ( S ) ( P , Q )
we see that, (cid:80) i ∈ S (cid:112) P ( i ) Q
( i ) ≥
P ( S )+ Q ( S )− d TV ( S ) ( P , Q ) .
Then we use the Cauchy-Schwarz inequality:(cid:88)
i ∈ S
[Equation]
A.2 Proof of Lemma
Lemma 2. [Equation] Proof. Let PQ be a distribution constructed from P and Q , where we ﬁrst sample j ∼ B Q and thensample i ∼ P S j , thus PQ ( i ) =
(cid:80) j ∈ [ k ] ∪{ 0 }
B Q ( j ) P S j ( i ) .
We know that if i ∈ S j , then i (cid:54)∈ S j (cid:48) forj (cid:48) (cid:54) =
j .
This allows us to simplify and write PQ ( i ) =
B Q ( j ) P S j ( i ) . Then,
[Equation]
Since d TV ( B P , B Q ) < ε 2 , we have d TV ( P , PQ ) <
ε 2 .
From the deﬁnition of TV, we have
[Equation]
We will need the following sets: R [Equation] { j : B P ( j [Equation]
Q [Equation] [Equation] [Equation]
B Q
[Equation]
P
[Equation]
From the triangle inequality we have d TV ( P , Q ) ≤
d TV ( P, PQ )
+ d TV ( PQ, Q ) .
We also knowthat d TV ( P, PQ ) <
ε 2 and d TV ( P , Q ) >
η .
Thus we have:
[Equation] j ) η − [Equation] (cid:88)
[Equation]
By deﬁnition, if j ∈ [ k ] \{ Bad ∪ R 1 ∪ R 2 } , then j has the property that d TV ( P S j , Q S j ) ≤ ε 1 .
Then,η − ε 2 < (cid:88)
j ∈{
[Equation]
R
[Equation]
[Equation]
[Equation]
+ (cid:88) j ∈
[Equation]
[Equation]
∪ Bad }
[Equation]
+ (cid:88) j ∈ Bad
B Q ( j ) η − ε 2 − ε 1 < B Q ( { 0 } ∪ R 1 ∪ R 2 ) + B Q ( Bad )
η − ε 2 − ε 1 − B Q ( { 0 } ∪ R 1 ∪ R 2 ) <
B Q ( Bad ) (2)
If i ∈ R 1 , then B P ( i ) > 2 B Q ( i ) , and thus B P ( i ) − B Q ( i ) > B Q ( i ) .
And thus,
B Q ( R 1 ) < (cid:88) i ∈ R( B P ( i ) − B Q ( i )) (3)
And if i ∈ R 2 , then B Q ( i ) > 5 B P ( i ) , and thus B Q ( i ) − B P ( i ) > 4 B P ( i ) , giving
B P ( R 2 ) < 1 4 (cid:88) i ∈ R( B Q ( i ) − B P ( i ))
B P ( R 2 ) + (cid:88) i ∈ R( B Q ( i ) − B P ( i )) <
5 4 (cid:88) i ∈ R( B Q ( i ) −
[Equation]
Since | S 0 | ≤ 2 n and all elements i ∈ S 0 satisfy P ( i ) ≤ 2 − k , we have B P (0) ≤ 2 n − k , where wesubstitute k = n + log 2 (100 /η ) to get
[Equation]
Then, B Q [Equation] [Equation]
[Equation]
B
[Equation]
R
[Equation] R 2 ) Using (3),( 4)) and (5) ≤ η 100 + (cid:88) [Equation]
[Equation]
i )
[Equation]
[Equation]
Here we partition the set Bad ∪ { 0 } into two sets Bad + and Bad − , where Bad + = { i ∈
Bad ∪{ 0 }| B P ( i ) ≥ B Q ( i ) } and similarly Bad − = { i ∈
Bad ∪ { 0 }| B P ( i ) <
B Q ( i ) } .
B Q ( Bad ) + B P ( Bad ) − 2 d TV ( Bad ) ( B Q , B P ) ≥ 2( B Q ( Bad )
− 2 d TV ( Bad ) ( B P , B Q ))
(From 2) >(cid:32) η − ε 2 − ε 1 − B Q ( { 0 } ∪ R 1 ∪ R 2 ) − (cid:88) i ∈ Bad | B P ( i ) − B Q ( i ) | (cid:33) (From 6) >  .
99 η − ε 2 − ε 1 − (cid:88) i ∈
{+R 
+}
∪ Bad
+
( B P ( i ) [Equation] i )) [Equation]
5(cid:88)
i ∈ R 2 ∪ Bad − ( B Q ( i ) − B P ( i )) 

And using the fact that d TV ( B P , B Q ) =
(cid:80) i : B P ( i ) ≤ B Q ( i ) ( B Q ( i ) − B P
( i ))
=
427 (cid:80) i : B P ( i ) >
B Q ( i ) ( B P ( i ) − B Q ( i )) =
ε 2 , and the fact that ∀ i ∈ R 1 B P ( i ) >
B Q ( i ) and∀ i ∈ R 2 B Q
( i ) >
B P ( i ) we have,
[Equation]
(cid:19)
A.3 Proof of Lemma
Recall that for all r, s ∈ [ m ] , E [ X r,s ]
= (cid:80)
b ∈ Bad B P ( b ) B Q ( b ) .
Then since X = (cid:80) r,s ∈ [ m ] X r,s ,
E [ X ] = (cid:80) r,s ∈ [ m ] E [ X r,s ]
= m 2 E [ X r,s ] .
Then for i, j, k, l ∈ [ m ] ,
• if i = k, j = l then E [ X i,j X k,l ] = (cid:80) b ∈ Bad B P ( b ) B Q ( b ) =
E [ X r,s ]
• if i = k, j (cid:54) =
l then E [ X i,j X k,l ] = (cid:80) b ∈ Bad B P ( b ) B 2 Q ( b )
• if i (cid:54) =
k, j = l then E [ X i,j X k,l ] = (cid:80)
b ∈ Bad B 2 P ( b ) B Q ( b )
• if i (cid:54) =
k, j (cid:54) =
l then E [ X i,j X k,l ] = (cid:0)(cid:80)
b ∈ Bad B P ( b ) B Q ( b ) (cid:1) 2 = E [ X r,s ]
E [ X 2 ] =
E   (cid:88) i,j,k,l ∈ [ m ]
X i,j X k,l   =
E   (cid:88) a (cid:54) =
c,b (cid:54) =
d i,j,k,l ∈ [ m ]
X i,j X k,l   + E   (cid:88) a = c,b (cid:54) =
d i,j,k,l ∈ [ m ]
X i,j X k,l   + E   (cid:88) a (cid:54) =
c,b = d i,j,k,l ∈ [ m ]
X i,j X k,l   + E   (cid:88)
a = c,b = d i,j,k,l ∈ [ m ]
X i,j X k,l   =
m 2 ( m − 1) 2 E
[ X r,s ] 2 + m 2 ( m − 1) (cid:32) (cid:88)
b ∈ Bad ( B P ( b ) + B Q ( b ))
B P ( i ) B Q ( i ) (cid:33) + m 2 E [ X r,s ] ≤ m 4 E [ X r,s ] 2 + m(cid:32) (cid:88)
b ∈ Bad ( B P ( b ) + B Q ( b ))
B P ( b ) B Q ( b ) (cid:33) + m 2 E [ X r,s ]
Then,
[Equation]
We will now focus on ﬁnding the maximum for ratio of summations:
[Equation]
(If b ∈ Bad then B P ( b ) /B Q ( b ) ∈ [5 − 1 , 2] )
[Equation]
(Using the monotonicity of (cid:96) p norms)
[Equation]
X
[Equation]
−
[Equation]
Thus,
[Equation]
r
[Equation]
(Since m 2 E [ X r,s ] ≥ 1 )
The Chebyshev bound from 2 tells us that Pr[ | X − E
[ X ] | < E [ X ]] > E [ X ] 2 / E
[ X 2 ] > 15 .
Thus,
[Equation]
(Since X takes only integer values)
A.4 Proof of Proposition
Proposition 5.
Let P , Q be distributions and let p ∼ P and q ∼ Q .
Then,1.
If d ∞ ( P , Q ) < ε then
[Equation]
ε 1 − ε
[Equation]
P
[Equation]
If d TV ( P , Q ) > ε 1 , then for 0 ≤ α < ε 1 , with probability at least ( d TV ( P , Q ) − α ) / 2 ,
Q ( p )
[Equation]
Proof.
If d ∞ ( P , Q ) < ε then
[Equation]
( p
[Equation]
and hence we show the ﬁrst part of the claim.
For the second part of the proof we introduce the some sets.
{+Let H 
+}
= { h | 1 ≤ Q ( h ) P ( h ) <
1 + α }and H 1 = { h | 1 + α ≤ Q ( h ) P ( h ) } and H
= h 0 ∪ H 1 .
Similarly deﬁne, L 0 = { (cid:96) | 1 − α < Q ( (cid:96) )
P ( (cid:96) ) < 1 } ,
L 1 = { (cid:96) | Q ( (cid:96) ) P ( (cid:96) ) ≤ 1 − α } and L = L 0 ∪ L 1 .
Now consider that we have a pair of samples, p ∼ P and q ∼ Q .
We know that either P ( L ) ≥ 1 / 2or P ( H ) > 1 / 2 .
P ( L ) ≥ 1 / 2 :
We see that Pr[ p ∈ L ] ≥ 1 / 2 .
Then from the deﬁnition of h 0 , Q ( h 0 ) − P ( h 0 ) < αand recall that Q ( H ) −P ( H ) =
d TV ( P , Q ) .
Thus we have that Q ( H 1 )
−P ( H 1 ) >
d TV ( P , Q ) − αand
hence Pr[ q ∈ H 1 ] > d TV ( P , Q ) − α .
We can now conﬁrm that q ∈ H 1 ∧ p ∈ L with probabilityat least ( d TV ( P , Q ) − α ) / 2 .
Then,
Q ( p )
Q ( p
[Equation]
[Equation]
<
P ( p )
P ( p ) + Q ( q ) (From P ( p ) > Q ( p ) ) <
P ( p )
P ( p ) + (1 + α ) P ( q ) (Since q ∈ H 1 )
P ( H ) > 1 / 2 : We see that Pr[ q ∈ H ] ≥ 1 / 2 .
Then we have that P ( L 0 )
−Q ( L 0 ) <
α and also that
P ( L ) − Q ( L ) =
d TV ( P , Q ) , we have that P ( L 1 ) − Q
( L 1 ) <
d TV ( P , Q ) − α .
Then, we deducethat probability at least ( d TV ( P , Q ) − α ) / 2 , q ∈ H ∧ p ∈
L 1 .
Then,
Q ( p )
Q ( p ) + Q ( q ) <
Q ( p )
Q ( p ) + P ( q ) (From P ( q ) < Q ( q ) ) <
P ( p )(1 − α )
P ( p )(1 − α ) + P ( q ) (Since p ∈ L 1 ) <
P ( p )
P ( p ) + (1 + α ) P ( q )
A.5 The outbucket subroutine
B Data missing from the main paper
[Table]
[Table]
[Table]
[Table]
[Table]
TOR s510_3_
TOR 7.sk_4_
TOR 56.sk_6_
TOR s820a_15_
TOR
ProjectService3.sk_12_
TOR s420_7_
TOR s832a_7_
TOR s420_new1_7_
TOR s420_15_
{+TOR s420_new_
+}
TOR s713_3_
TOR s526a_15_
TOR s1196a_7_
TOR 81.sk_5_
TOR s420_new_3_
TOR s349_15_
TOR s344_15_
TOR s713_7_
TOR 77.sk_3_
TOR s420_new_15_
TOR s832a_3_
TOR UserServiceImpl.sk_8_ TOR 19.sk_3_ TOR s953a_7_
TOR
[Table] [Table]
[Table]
[Table]
s344_7_
TOR s1196a_3_
TOR s444_7_
TOR 51.sk_4_
TOR 57.sk_4_
TOR 53.sk_4_
TOR s832a_15_
TOR s953a_3_
TOR 63.sk_3_
TOR s526_15_
TOR 110.sk_3_
TOR s349_3_
TOR s820a_3_
TOR s1196a_15_
TOR 10.sk_1_
TOR s1238a_7_ TOR s420_new1_15_
TOR s344_3_
TOR s953a_15_
TOR s526a_7_
TOR 80.sk_2_
TOR 32.sk_4_
TOR s820a_7_
TOR s382_15_
TOR 17.sk_3_
TOR s382_7_
TO - [Table]