Sampling from Log-Concave Distributions with Inﬁnity-Distance Guarantees
Anonymous Author(s)
AfﬁliationAddress email
Abstract
For a d -dimensional log-concave distribution ﬁ ( ◊ ) Ã e ≠ f ( ◊ ) constrained to a convex body K , the problem of outputting samples from a distribution ‹ which isÁ -close in inﬁnity-distance sup ◊ œ K | log ‹ ( ◊ ) ﬁ ( ◊ ) | to ﬁ arises in differentially privateoptimization.
While sampling within total-variation distance Á of ﬁ can be doneby algorithms whose runtime depends polylogarithmically on 1 Á , prior algorithmsfor sampling in Á inﬁnity distance have runtime bounds that depend polynomiallyon 1 Á .
We bridge this gap by presenting an algorithm that outputs a point Á -closeto ﬁ in inﬁnity distance that requires at most poly(log 1 Á , d ) calls to a membershiporacle for K and evaluation oracle for f , when f is Lipschitz.
Our approach departs from prior works that construct Markov chains on a 1 Á 2 -discretization of Kto achieve a sample with Á inﬁnity-distance error, and present a method to directlyconvert continuous samples from K with total-variation bounds to samples withinﬁnity bounds.
This approach also allows us to obtain an improvement on thedimension d in the running time for the problem of sampling from a log-concavedistribution on polytopes K with inﬁnity distance Á , by plugging in TV-distancerunning time bounds for the Dikin Walk Markov chain.
Introduction
The problem of sampling from a log-concave distribution is as follows: For a convex body K ™
R d and a convex function f : K æ R , output a sample ◊ from the distribution ﬁ ( ◊ ) Ã e ≠ f ( ◊ ) .
This is a basic problem in computer science, statistics and machine learning, with applicationsto optimization and integration [1, 29], Bayesian statistics [39], reinforcement learning [6] anddifferential privacy [32, 20, 2, 24].
Sampling exactly from ﬁ is known to be computationally hard formost interesting cases of K and f [16] and, hence, the goal is to output samples from a distribution‹ that is at a small (speciﬁed) “distance” to ﬁ .
For applications such as computing the integral ofﬁ , bounds in the total variation (TV) distance [1] or KL divergence (which implies a TV bound) aresufﬁcient.
In applications such as computing the expectation of a Lipschitz function with respectto ﬁ , Wasserstein distance may also be sufﬁcient.
In differentially private optimization [32, 20, 2,19, 24], one requires bounds on the stronger inﬁnity-distance – d Œ ( ‹ , ﬁ ) :
= sup ◊ œ K | log ‹ ( ◊ ) ﬁ ( ◊ ) | – toguarantee pure differential privacy, and TV, KL, or Wasserstein bounds are insufﬁcient; see [14].
Pure differential privacy (DP) is the strongest notion of DP and has been extensively studied (see e.g.the survey [14]).
It has advantages over weaker notions of differential privacy.
E.g., when privacyof “groups" of individuals (rather than just single individuals) must be preserved, any mechanismwhich is (pure) Á -DP (with respect to single individuals), is also k Á -DP with respect to subsets ofk individuals.
Motivated by applications to differential privacy, we study the problem of designingefﬁcient algorithms to output samples from a distribution ‹ which is Á -close in d Œ to ﬁ .
Related works.
Several lines of work have designed Markov chains that generate samples fromdistributions that are close to a given log-concave distribution.
These results differ in both their assumptions on the log-density and its support, as well as the distance used to measure closeness.
Oneline of work includes bounds for sampling from a log-concave distribution on a compactly supportedconvex body within TV distance O ( ” ) , including results with running time that is polylogarithmicin 1 ” [1, 30, 29, 34] (as well as other results which give a running time bound that is polynomial in 1 ”[18, 17, 5, 4]).
In addition to assuming access to a value oracle for f , some Markov chains just needaccess to a membership oracle for K [1, 30, 29], while others assume that K is a given polytope:{ ◊ œ R d : A ◊ Æ b } [23, 33, 35, 34, 26].
They often also assume that K is contained in a ball ofradius R and contains a ball of smaller radius r .
Many of these results assume that the target logconcave distribution satisﬁes a “well-rounded” condition which says that the variance of the targetdistribution is  ( d ) [30, 29], or that it is in isotropic position (all eigenvalues of its covariance matrix are  (1) )
[25]; when applied to log-concave distributions that are not well-rounded or isotropic,these results require a “rounding” pre-processing procedure to ﬁnd a linear transformation whichmakes the target distribution well-rounded or isotropic.
Finally, it is often assumed that the function f is such that f is L -Lipschitz or — -smooth [34], including works handling the widely-studiedspecial case when f is uniform on K where L = — =
(see e.g. [28, 23, 33, 35, 26, 8, 21]).
Another line of work gives sampling algorithms with bounds on the distance to the target density ﬁin terms of Wasserstein distance [13, 11], KL divergence [40, 12], and Renyi divergence [36].
Incontrast to works which assume access to an oracle for the value of f , many of these results insteadassume access to an oracle for the gradient of f and require the log-density to be L -Lipschitz or— -smooth on all of R d (or on, e.g., a cube containing K ) for some L, — > 0 .
However, as notedearlier, bounds in the Wasserstein distance, KL divergence, and – -Renyi divergence (for – < Œ )also do not imply bounds on the inﬁnity distance, and the running time bounds provided by theseworks are polynomial in 1 Á .
(See also Appendix A for additional discussion and challenges.)
Among prior works that give algorithms with bounds on d Œ , [20] applies the grid walk Markovchain of [1] to sample from a uniform distribution on a convex body.
[2] extends the approachof [20] to log-Lipschitz log-concave distributions.
Unlike the TV-distance case where algorithmswhose running time depends logarithmically on the error are known (e.g., [1, 29, 34]), the bestavailable bounds for sampling within O ( Á ) inﬁnity-distance [20, 2] have runtime that is polynomialin 1 Á and a relatively large polynomial in d .
Our contributions.
We present a new approach to output samples, which come with d Œ bounds,from a log-concave and log-Lipschitz distribution constrained to a a convex body.
Speciﬁcally,when K :=
{ ◊ : A ◊ Æ b } is a polytope (where one is given A and b ) our main result (Theorem2.1) guarantees samples from a distribution that is within O ( Á ) error in d Œ and whose runtime depends logarithmically on 1 Á compared to the polynomial dependence of [2].
Our approach departsfrom prior works that construct Markov chains on a 1 Á 2 -discretization of K to achieve a samplewith Á inﬁnity-distance error, and we present a method (Algorithm 1) to directly convert continuoussamples from K with total-variation bounds to samples with inﬁnity bounds (Theorem 2.2).
Thiscontinuous-space approach also allows us to obtain an improvement on the dimension d in the running time when K is a polytope by plugging in TV-distance running time bounds for the Dikin Walk
Markov chain of [34].
As immediate applications, we obtain faster algorithms for differentiallyprivate empirical risk minimization (Corollary 2.4) and low rank approximation (Corollary 2.5).
Results
Let B ( v, s ) :
= { z œ R d : Î z ≠ v Î 2 Æ s } and Ê denote the matrix-multiplication constant.
Theorem 2.1 (Main result)
There exists an algorithm which, given Á , L, r, R > 0 , A œ R m ◊ d ,b œ R m that deﬁne a polytope K :=
{ ◊ œ R d : A ◊ Æ b } contained in a ball of radius R , a pointa œ R d such that K contains a ball B (0 , r ) of smaller radius r , and an oracle for the value of aconvex function f : K æ R d , where f is L -Lipschitz, and deﬁning ﬁ to be the distribution ﬁ Ã e ≠ f ,outputs a point from a distribution ‹ such that d Œ ( ‹ , ﬁ ) < Á .
Moreover, with very high probability 1 ,this algorithm takes O ( T ) function evaluations and O ( T ◊ md Ê ≠ 1 ) arithmetic operations, where
T = O (( m 2 d 3 + m 2 dL 2 R 2 ) ◊ [ LR + d log( Rd + LRdr Á )]) .
In comparison to the polynomial in 1 Á runtime bounds of [2], Theorem 2.1 guarantees a runtime thatis logarithmic in 1 Á , and also improves the dependence on the dimension d , in the setting where Kis a polytope.
Speciﬁcally, [2] show that the number of steps of the grid walk to sample from ﬁwith inﬁnity-distance error at most Á is O ( 1 Á 2 ( d 10 + d 6 L 4 R 4 )
◊ polylog( 1 Á , 1 r , R, L, d ))
(Lemma 6.5in the Arxiv version of [2]).
When applying their algorithm to the setting where f is constrainedto a polytope K = { x œ R d : Ax Æ b } , each step of their grid walk Markov chain requires 93 computing a membership oracle for K and the value of the function f .
The membership oraclecan be computed in O ( md ) arithmetic operations.
Thus the bound on the number of arithmeticoperations for each step of their grid walk is O ( md ) (provided that each function evaluation takesat most O ( md ) arithmetic operations).
Thus the bound on the number of arithmetic operations toobtain a sample from ﬁ is O ( 1 Á 2 ( md 11 + md 7 L 4 R 4 )
◊ polylog( 1 Á , 1 r , R, L, d )) .
Thus, Theorem 2.1improves on this bound by a factor of roughly 1 Á 2 m 3 d 8 ≠ Ê .
For example, when m = O ( d ) , as maybe the case in differentially private applications, the improvement is 1 Á 2 d 5 ≠ Ê .
We note that the bounds of [2] also apply in the more general setting where K is a convex bodywith membership oracle.
One can extend our bounds to achieve a runtime that is logarithmic in 1 Á(and polynomial in d, L, R ) in the more general setting where K is a convex body with membershiporacle; we omit the details (see Remark 2.3).
Moreover, we also note that while there are several results which achieve O ( ” ) TV bounds in timelogarithmic in 1 ” , TV bounds do not in general imply O ( Á ) bounds on the KL or Renyi divergence,or on the inﬁnity-distance, for any ” > 0 .
On the other hand, an Á -inﬁnity-distance bound doesimmediately imply a bound of Á on the KL divergence D KL , and – -Renyi divergence D – , since
D KL ( µ, ﬁ ) Æ d Œ ( µ, ﬁ ) and D – ( µ, ﬁ ) Æ d Œ ( µ, ﬁ ) for any – > 0 and any pair of distributions µ, ﬁ .
Thus, under the same assumptions on K and f , Theorem 2.1 implies a method of sampling from a
Lipschitz concave log-density on K with Á KL and Renyi divergence error in a number of arithmeticoperations that is logarithmic in 1 Á , with the same bound on the number of arithmetic operations.
The polynomial dependence on 1 Á in [2] is due to the fact that they rely on a discrete-space Markovchain [1], on a grid with cells of width w =
O ( Á L Ô d ) , to sample from ﬁ within O ( Á ) inﬁnity-distance.
Since their Markov chain’s runtime bound is polynomial in w ≠ 1 , they get a runtime bound forsampling within O ( Á ) inﬁnity-distance that is polynomial in 1 Á .
The proof of Theorem 2.1 bypassesthe use of discrete grid-based Markov chains by introducing Algorithm 1 which transforms anysample within ” =
O ( Á e ≠ d ≠ nLR ) TV distance of the distribution ﬁ Ã e ≠ f on the continuous set
K (as opposed to a discretization of K ), into a sample within O ( Á ) inﬁnity-distance from ﬁ .
Thisallows us to make use of a continuous-space Markov chain, whose step size is not restricted to a gridof width O ( Á L Ô d ) and is instead independent of Á , to obtain a sample within O ( Á ) inﬁnity-distancefrom ﬁ in time that is logarithmic in 1 Á .
Theorem 2.2 (Main technical contribution: Converting TV bounds to inﬁnity-distance bounds)
There exists an algorithm (Algorithm 1) which, given Á , r, R, L > 0 , a membership oracle for aconvex body K contained in a ball of radius R and containing a ball B (0 , r ) , and an oracle whichoutputs a point from a distribution µ which has TV distance ” Æ O ( Á ◊ ( R ( d log( R / r )+ LR ) 2 Á r )
≠ d e ≠ LR )from a distribution ﬁ Ã e ≠ f where f : K æ R is an L -Lipschitz function (see Appendix B for theexact values of ” and related hyper-parameters), outputs a point ˆ ◊ œ K such that the distribution
‹of ˆ ◊ satisﬁes d Œ ( ‹ , ﬁ ) Æ Á .
Moreover, with very high probability 3 , this algorithm ﬁnishes in O (1)calls to the sampling and membership oracles, plus O ( d ) arithmetic operations.
To the best of our knowledge Theorem 2.2 is the ﬁrst result which for any Á , L, r, R > 0 , whenprovided as input a sample from a continuous-space distribution on a convex body K within some
TV distance ” = ” ( Á , r, R, L ) > 0 from a given L -log-Lipschitz distribution ﬁ on K , where Kis contained in a ball of radius R and containing a ball of smaller radius r , outputs a sample with
Algorithm 1 ﬁnishes in · calls to the sampling and membership oracles, plus O ( · d ) arithmetic operations, where E [ · ] Æ 3 and P ( · Ø t ) Æ ! "
t for all t Ø 0 and · Æ 5 d log( Rr ) + 5 LR + 2 w.p. 1. distribution within inﬁnity-distance O ( Á ) from ﬁ .
This is in contrast to previous works [2] (see also[20] which applies only to the special case where ﬁ is the uniform distribution on K ) which insteadrequire as input a sample with bounded TV distance from the restriction of ﬁ on a discrete grid on
K , and then convert this discrete-space sample into a sample within inﬁnity-distance O ( Á ) from thecontinuous-space distribution ﬁ : K æ R .
8 end
Remark 2.3 (Extension to convex bodies with membership oracles)
We note that Theorem 2.1can be extended to the general setting where K is an arbitrary convex body in a ball of radius
R and containing a ball of smaller radius r , and we only have membership oracle access to K .
Namely, one can plug in the results of [29] to our Theorem 2.2 to generate a sample from a LLipschitz concave log-density on an arbitrary convex body K in a number of operations that is(poly)-logarithmic in 1 Á , 1 r and polynomial on d, L, R .
We omit the details.
Applications to differentially private optimization.
Sampling from distributions with O ( Á )inﬁnity-distance error has many applications to differential privacy.
Here, the goal is to ﬁnd a randomized mechanism h : D n æ R which, given a dataset x œ D n consisting of n datapoints,outputs model parameters
ˆ ◊ œ R in some parameter space R , which minimize a given (negative)utility function f
( ◊ , x ) , under the constraint that the output ˆ ◊ preserves the pure Á -differential privacy of the data points x .
A randomized mechanism h : D n æ R is said to be Á -differentiallyprivate if for any datasets x, x Õ œ D which differ by a single datapoint, and any S ™ R , we have 153 that
P ( h ( x ) œ S ) Æ e Á P ( h ( x Õ ) œ S ) ; see [14].
As one application of Theorem 2.1, we consider the problem of ﬁnding an (approximate) minimumˆ ◊ of an empirical risk function f : K ◊ D n æ R under the constraint that the output ˆ ◊ is Ádifferentially private, where f ( ◊ , x ) :
= q ni =1 ¸ i ( ◊ , x i ) .
Following [2], we assume that the ¸ i ( · , x )are L -Lipschitz for all x œ D n , i œ N , for some given L > 0 .
In this setting [2] show that theminimum ERM utility bound under the constraint that ˆ ◊ is pure Á -differentially private, E ˆ ◊ [ f (ˆ ◊ , x )]
≠min ◊ œ K f ( ◊ , x ) =  ( dLR Á ) , is achieved if one samples ˆ ◊ from the exponential mechanism ﬁ Ãe ≠ Á 2 LR f with inﬁnity-distance error at most O ( Á ) .
Plugging Theorem 2.1 into the framework ofthe exponential mechanism, we obtain a pure Á -differentially private mechanism which achieves theminimum expected risk (Corollary 2.4, see Section C.1 for a proof).
Corollary 2.4 (Differentially private empirical risk minimization)
There exists an algorithmwhich, given Á , L, r, R > 0 , A œ R m ◊ d , b œ R m that deﬁne a polytope K :=
{ ◊ œ R d : A ◊ Æ b }contained in a ball of radius R and containing a ball B (0 , r ) of smaller radius r , and a convexfunction f
( ◊ , x ) :=
q ni =1 ¸ i ( ◊ , x i ) , where each ¸ i : K æ R is L -Lipschitz, outputs a random pointˆ
◊ œ K which is pure Á -differentially private and satisﬁes E
ˆ ◊ [ f (ˆ ◊ , x )]
≠ min ◊ œ K f ( ◊ , x ) Æ O ( dLR Á ) .
Moreover, this algorithm takes at most T ◊ md Ê ≠ 1 arithmetic operations plus T evaluations of thefunction f ,
where T =
O
(( m 2 d 3 + m 2 dn 2 Á 2 ) ◊ ( Á n + d )log 2 ( nRdr Á )) .
Corollary 2.4 improves on the previous bound [2] of O (( 1 Á 2 ( m + n ) d 11 + Á 2 n 4 ( m + n ) d 7 ) ◊polylog( nRdr Á ))) arithmetic operations by a factor of roughly max( d 8 ≠ Ê Á 2 m 2 , 1 Á m 2 nd 5 ) , in the settingwhere the ¸ i are L -Lipschitz on a polytope K and each ¸ i can be evaluated in O ( d ) operations.
See
Appendix C.1 for a proof of this corollary.
As another application, we consider the problem of ﬁnding a low-rank approximation of a samplecovariance matrix  = q ni =1 u
i u € i where the datapoints u
i œ R d satisfy Î u i Î Æ 1 , in a differen- 176 tially private manner.
Given any k > 0 , the goal is to ﬁnd a (random) rank- k projection matrix Pwhich maximizes the average variance E P [ È  , P Í ] of the matrix  (also reffered to as the utilityof P ), under the constraint that the mechanism which outputs the matrix P is Á -differentially private. This problem has many applications to statistics and machine learning, including differentiallyprivate principal component analysis (PCA) [7, 3, 15, 24].
When privacy is not a concern, the solution P which maximizes the variance is just a projectionmatrix onto the subspace spanned by top-
k eigenvectors of  , and the maximum variance satisﬁesÈ  , P Í = q ki =1 ⁄
i , where ⁄ 1 Ø · · · Ø ⁄ d > 0 denote the eigenvalues of  .
However, when privacy 184 is a concern, there is a tradeoff between the desired privacy level Á and the utility E P [ È  , P Í ] , and 185 the maximum utility E P
[ È  , P Í ] one can achieve decreases with the privacy parameter Á .
The bestcurrent utility bound for an Á -differentially private low rank approximation algorithm was achievedin [24], who show that one
can ﬁnd a pure Á -differentially private random rank-
k projection matrix P such that E P [ È  , P Í ] Ø (1 ≠ ” )
q ki =1 ⁄ i whenever q ki
=1 ⁄ i Ø  ( dk Á” log 1 ” ) for any ” > 0 . Togenerate the matrix P , their algorithm generates a sample, with inﬁnity-distance error O ( Á ) , froma Lipschitz concave log-density on a polytope, and transforms this sample into a projection matrix P .
The sampling algorithm used in [24] has a bound of poly( 1 Á , d, ⁄ 1 ≠ ⁄ d ) arithmetic operationsand they leave as an open problem whether this can be improved from a polynomial dependence on1 Á to a logarithmic dependence on 1 Á .
Corollary 2.5 shows that a direct application of Theorem 2.1resolves this problem.
(See Section C.2 for a proof.)
Corollary 2.5 (Differentially private low rank approximation)
There exists an algorithm which,given a sample covariance matrix  = q ni =1 u
i u € i for datapoints
u i œ R d satisfying Î u
i
Î
Æ 1 , 197 its eigenvalues ⁄ 1 Ø . . .
⁄
d > 0 , an integer k , and Á , ” > 0 , outputs a random rank-k symmetric projection matrix P such that P is Á -differentially private and satisﬁes the utility bound E P [ È  , P Í ] Ø (1 ≠ ” )
q ki =1 ⁄
i (  ) whenever q ki =1 ⁄
i (  ) Ø C dk Á” log 1 ” for some universal constant C > 0 .
Moreover the number of arithmetic operations is logarithmic in 1 Á and polynomial ind and ⁄ 1 ≠ ⁄ d .
Proof Overviews
Given any Á , and a function f : R d æ R , the goal is to sample from a distribution ﬁ ( ◊ ) Ã e ≠ f ( ◊ ) ,constrained to a d -dimensional convex body K with inﬁnity-distance error at most O ( Á ) in a numberof arithmetic operations that is logarithmic in 1 Á .
We assume that K is contained in a ball of someradius R > 0 and contains a ball of some radius r > 0 , and f is L -Lipschitz.
In addition wewould also like our bounds to be polylogarithmic in 1 r , and polynomial in d, L, R with a lowerorder dependence on the dimension d than currently available bounds for sampling from Lipschitzconcave log-densities on a polytope in inﬁnity-distance [2].
We note that since whenever K iscontained in a ball of radius R and contains a ball B (0 , r ) of smaller radius r , we also have that
B (0 , r ) ™ K ™ B (0 , 2 R ) , without loss of generality, we may assume that B (0 , r ) ™ K ™ B (0 , R )as this would only change the bounds provided in our main theorems by a constant factor.
The main ingredient in the proof of Theorem 2.1 is Theorem 2.2 that uses Algorithm 1 to transforma TV-bounded sample into a sample from ﬁ with error bounded in d Œ .
Subsequently, we invoke
Theorem 2.1 when K is given as a polytope K :=
{ x œ R d : Ax Æ b } and plug in the Dikin
Walk Markov chain of [34] which generates independent samples from ﬁ with bounded TV error.
We ﬁrst present an overview of the proof of Theorem 2.2.
(The full proof has been omitted to spacerestrictions and presented in Appendix B.)
The proof of Theorem 2.1 is presented in Section 3.2.
3.1 Converting samples with TV bounds to inﬁnity-distance bounds
proof of Theorem 2.
Impossibility of obtaining log-dependence on inﬁnity-distance via grid walk.
One approach isto observe that if e ≠ f has support on a discrete space S with at most | S | points, then any ‹ suchthat Î ‹ ≠ ﬁ Î TV Æ Á also satisﬁes d Œ ( ‹ , ﬁ ) Æ 2 | S | max z œ S e ≠ f ( z ) min z œ S e ≠ f ( z ) ◊ Á for any Á Æ min z œ S ﬁ ( z ) .
This suggests forming a grid G over K , then using a discrete Markov chain to generate a sample ◊within O ( Á ) TV distance of the discrete distribution ﬁ G Ã e ≠ f with support on the grid G , and thendesigning an algorithm which takes as input ◊ and outputs a point with bounded inﬁnity-distance tothe continuous distribution ﬁ .
This approach was used in [20] in the special case when ﬁ is uniformon K , and then extended by [2] to log-Lipschitz log-concave distributions.
In their approach, [2]ﬁrst run a “grid-walk” Markov chain on a discrete grid in a cube containing K .
They then apply thebound from [1] which says that the grid walk obtains a sample Z within TV distance O ( ” ) from thedistribution Ã e ≠ f (restricted to the grid) in time that is polylogarithmic in 1 ” and quadratic in a ≠ 1 ,where a is the distance between neighboring grid points.
Since their grid has size | S | =
(( Ra ) d ) ,a TV distance of O ( ” ) automatically implies an inﬁnity-distance of O ( ” c | S | ) , where c is the ratioof the maximum to the minimum probability
mass satisfying c
Æ e LR since f is L -Lipschitz on
K ™ B (0 , R ) .
Thus, by using the grid walk to sample within TV-distance ” =
O ( Á | S | c ) from thediscrete distribution ﬁ G , they obtain a sample Z which also has inﬁnity-distance O ( Á ) from ﬁ G .
Finally, to obtain a sample from the distribution ﬁ Ã e ≠ f on the continuous space K , they samplea point uniformly from the “grid cell”
[ Z ≠ a, Z + a ] d centered at Z .
Since f is L -Lipschitz, theratio e ≠ f ( Z )
e ≠ f ( w ) is bounded by O ( Á ) for all w in the grid cell [ Z ≠ a, Z + a ] d as long as a =
O ( Á L Ô d ) ,implying that the sample is an inﬁnity-distance of O ( Á ) from ﬁ Ã e ≠ f .
However, since the runningtime bound of the grid walk is quadratic in a ≠ 1 , the grid coarseness a
=
O
( Á L Ô d ) needed to achieve
O ( Á ) inﬁnity-distance from ﬁ leads to a running time bound which is quadratic in 1 Á .
To get around this problem, rather than relying on the use of a discrete-space Markov chain such asthe grid walk to sample within O ( Á ) inﬁnity-distance from ﬁ , we introduce an algorithm (Algorithm1) which transforms any sample within ” =
O ( Á e ≠ d ≠ LR ) TV distance from the distribution ﬁ Ãe ≠ f on the continuous space K (as opposed to a grid-based discretization of K ), into a sample within
O ( Á ) inﬁnity-distance from ﬁ .
This allows us to make use of a continuous-space Markov chain, suchas the Dikin walk of [34], whose step-size is not restricted by a grid of coarseness w =
O ( Á L Ô d )and instead is independent of Á , in order to generate a sample within O ( Á ) inﬁnity-distance from ﬁin runtime that is logarithmic in 1 Á .
Converting continuous space TV-bounded samples to inﬁnity-distance bounded samples.
Asdiscussed in Section 1, there are many Markov chain results which allow one to sample from a logconcave distribution on K with error bounded in weaker metrics such as total variation, Wasserstein,or KL divergence.
However, when sampling from a continuous distribution, bounds in these metricsdo not directly imply bounds in inﬁnity-distance.
And techniques used to prove bounds in weakermetrics do not easily extend to methods for bounding the inﬁnity-distance; see Section A.1.
Convolving with continuous noise.
As a ﬁrst attempt, we consider the following simple algorithm:sample a point ◊ ≥ µ from a distribution µ with total variation error Î µ ≠ ﬁ Î TV Æ O ( Á ) .
Since f 258 is L -Lipschitz, for any  < Á
L and any ball B ( z,  ) in the  -interior of K (denoted by int  ( K ) ;see Deﬁnition B.1), we can obtain a sample from a distribution ‹ such that log( ‹ ( z ) µ ( z ) )
Æ Á for allz œ int  ( K ) by convolving µ with the uniform distribution on the ball B (0 ,  ) .
Sampling from thisdistribution ‹ can be achieved by ﬁrst sampling ◊ ≥ µ and then adding noise › ≥ Unif( B (0 ,  )) tothe sample ◊ .
Unfortunately, this simple algorithm does not allow us to guarantee that log( ‹ ( z ) µ ( z ) )
Æ Á at pointsz / œ int  ( K ) which are a distance less than  from the boundary of K .
To see why, supposethat K = [0 , 1] d is the unit cube, that f is constant on K , and consider a point w = (1 , . .
, 1) atthe corner of the cube K .
In this case we could have that ‹ ( z ) Æ
≠ d ﬁ ( z ) for all z in some ballcontaining w , and hence d Œ ( ‹ , ﬁ ) =
sup | log( ‹ ( z ) ﬁ ( z ) ) | Ø d log(2) , no matter how small we make  .
Stretching the convex body to handle points close to the boundary.
To get around this problem, wewould like to design an algorithm which samples from some distribution ‹ such that | log ‹ ( z ) ﬁ ( z ) | Æ Á for all z œ K , including at points z near the corners of K .
Towards this end, we ﬁrst consider thespecial case where K is itself contained in the  -interior of another convex body K Õ , the functionf : K æ R extends to an L -Lipschitz function on K Õ (also referred to here with slight abuse ofnotation as f ) and where we are able to sample from the distribution
Ã e ≠ f on K Õ with O ( Á ) totalvariation error.
If we sample ◊ ≥ e ≠ f on K Õ with total variation error O ( ” ) where ” Æ Á e ≠ d log( R ) ,add noise › ≥ Unif( B (0 ,  )) to ◊ for  = ” LR , and then reject ◊ + › only if it is not in K , weobtain a sample whose distribution is O ( Á ) from the distribution Ã e ≠ f on K in inﬁnity-distance.
However, we would still need to deﬁne and sample from such a convex body K Õ , and to make surethat K Õ is not too large when compared to K ; otherwise the samples from the distribution Ã e ≠ fon K Õ may be rejected with high probability.
Moreover, another issue we need to deal with is that fmay not even be deﬁned outside of K .
To get around these two problems, in Algorithm 1, we begin by taking as input a point ◊ ≥ µ sampledfrom some distribution µ supported on K where Î µ ≠ ﬁ Î TV Æ ” for some ” Æ
Á e ≠ d log( R ) , and addnoise › ≥ unif( B (0 ,  r )) in order to sample from a distribution ˆ µ which satisﬁes --- log ˆ µ ( z ) ﬁ ( z ) --- Æ Áfor all z œ int  r ( K ) .
Here r is the radius of the small ball contained in K ; the choice of radius  rfor the noise distribution is because we will show in the following paragraphs that to sample fromthe distribution ﬁ on K with inﬁnity-distance error Á it is sufﬁcient sample a point in the  r -interiorof K and to then apply a “stretching" operation to K .
We still need a method of sampling within O ( Á ) inﬁnity-distance error of ﬁ on all of K , including in the region K \ int  r ( K ) near the boundary of K .
Towards this end, after Algorithm 1 generates a point Z
= ◊ + › from the above-mentioned distribution ˆ µ , it then multiplies
Z by 1 1 ≠  and returns the resulting point
ˆ ◊ := 1
≠  Z if it is K , in other words, if Z œ (1 ≠  ) K .
If we can show that (1 ≠  ) K ™ int  r ( K ) , then this would imply that --- log ˆ µ ( z ) ﬁ ( z ) --- Æ Á for all z œ (1 ≠  ) K , and hence that the distribution of ˆ ‹ of the returned point
ˆ ◊ satisﬁes
[Equation]
for all z œ K .
Since f is L -Lipschitz we have
[Equation]
--- = O ( Á ) for all ◊ œ K , and hence wewould then have that the distribution ˆ ‹ of the point ˆ ◊ returned by Algorithm 1 satisﬁes
[Equation]
However, for (1) to hold, we still need to show that (1 ≠  ) K ™ int  r ( K ) (proved in Lemma B.1).
In other words, we would like to show that for any point Z œ (1 ≠  ) K , there is a ball B ( Z,  r )centered at Z of radius  r contained in K .
To show this fact, it is sufﬁcient to consider the convexhull C of B (0 , r ) ﬁ Ó 1 1 ≠  Z Ô ™
K , and show that it contains the ball B ( Z,  r ) .
Towards thisend, we make the following geometric construction: we let p be a point such that the line p ˆ ◊ istangent to B (0 , r ) , and q the point on p ˆ ◊ which minimizes the distance Î q ≠ Z Î 2 (see Figure 1 inthe Appendix).
Since \ 0 p ˆ ◊ and \
Zq ˆ ◊ are both right angles, we have that the triangles
p ˆ ◊ and Zq ˆ ◊are similar triangles, and hence that Î Z ≠ q Î 2 r
= Î Z ≠ ˆ ◊ ÎÎ ˆ ◊ ≠ {+
+} Î 2 .
In other words,
[Equation]
implying a ball of radius  r centered at Z is contained in C ™ K , and hence (1 ≠  ) K ™ int  r ( K ) .
Bounding the inﬁnity distance error.
To complete the bound on the inﬁnity-distance of the distribution ˆ ‹ of the point returned by Algorithm 1 to the target distribution ﬁ , we must showboth a lower bound (Lemma B.5) and an upper bound (Lemma B.6) on the ratio ˆ ‹ ( ◊ ) ﬁ
( ◊ ) at everypoint ◊ œ K .
Both the upper and lower bounds are necessary to bound the inﬁnity-distanced Œ (ˆ ‹ ( ◊ ) , ﬁ ( ◊ ))
=
sup ◊ œ K | log ˆ ‹ ( ◊ ) ﬁ ( ◊ ) | .
Both Lemmas B.5 and B.6 require the input point to have TV error ” <
Á ( R  r )
≠ d e ≠ LR .
The term( R  r ) ≠ d is a lower bound on the ratio of the volume of K to the volume of the smoothing ball
B (0 ,  r ) ; this bound holds since K is contained in a ball of radius R .
The term e ≠ LR is a lowerbound on the ratio
[Equation] K ﬁ ( w ) [Equation] œ K ﬁ ( w ) of the minimum value of the density ﬁ to the maximum value of ﬁat any two points in K ; this bound holds since f is L -Lipschitz.
The above choice of ” ensures that in any ball B ( z,  r ) with center z in the  r -interior of K ,the distribution µ of the input point, which satisﬁes Î µ ≠ ﬁ Î TV Æ ” , will have between e ≠ Á and 311 e Á times the probability mass which the target distribution ﬁ has inside the ball B ( z,  r ) .
Thus,when the distribution µ is smoothed by adding noise uniformly distributed on a ball of radius  r ,the smoothed distribution ˜ ‹ ( ◊ ) is within e ≠ Á and e Á times the target probability density ﬁ ( ◊ ) at anypoint ◊ in the  r -interior of K , allowing us to bound the inﬁnity distance error of the smootheddistribution ˜ ‹ at any point ◊ in the  r -interior of K .
We then apply this fact, together with Lemma
B.1 which says that for any point ◊ œ K the point (1 ≠  ) ◊ is in the  r -interior of K , to bound thedistribution ‹ of the output point (after the stretching operation) as follows,
[Equation]
Here our choice of hyperparameter  Æ Á max( d,LR ) ensures that (1 ≠  ) d
=  (1) and, since f is
L -Lipschitz, that ﬁ ((1 ≠  ) ◊ ) Ø e ≠ Á ﬁ ( ◊ ) .
This proves the lower bound (Lemma B.5).
The proofof the upper bound (Lemma B.6) follows in a similar way as equation (2) but with the inequalitiesgoing in the opposite direction.
Bounding the number of iterations and concluding the proof of Theorem 2.2.
We still need to dealwith the problem that the point ˆ ◊ may not be accepted.
If this occurs, roughly speaking, we repeat theabove procedure until a point ˆ ◊ with distribution ˆ ‹ is accepted.
To bound the number of iterations,we show that ˆ ◊ is in K with high probability.
Towards this end, we ﬁrst use the facts that f is LLipschitz and K ™ B (0 , R ) , to show that the probability a point sampled from ﬁ Ã e ≠ f lies inside(1 ≠  ) K is at least (1 ≠  )
d e ≠ L  R Ø 910 (Lemma B.2).
Lemma B.2 says that if you stretch thepolytope by a factor of 1 1 ≠  , then most of the volume of the stretched polytope ( 1 1 ≠  ) K remainsinside the original polytope K .
The term (1 ≠  ) d is just the ratio of the volume of (1 ≠  ) K to thevolume of K .
And, since f is L -Lipschitz, the term e ≠ L  R bounds the ratio ﬁ ( ◊ ) ﬁ ( 1≠  ◊ ) of the targetdensity at any point ◊ œ K to the value of ﬁ at the point 1 1 ≠  ◊ to which the stretching operationtransports ◊ , whenever 1 1 ≠  ◊ œ K .
The choice of hyperparameter  Æ Á max( d,LR ) ensures that theacceptance probability (1 ≠  ) d e ≠ L  R guaranteed by Lemma B.2 is at least 910 .
Since the convex body (1 ≠  ) K contains the ball B (0 , r 2 ) , applying Lemma B.1 a second time(this time to the convex body (1 ≠  ) K ) we get that (1 ≠ 3  ) K ™ int  r ((1 ≠  ) K ) .
Thus, by
Lemma B.2 we have that ◊ lies inside int  r ((1 ≠  ) K ) with probability at least 910 ≠ ” Ø 810 (as 337 ◊ is sampled from ﬁ with TV error Æ ” ).
Therefore, since › ≥ B (0 ,  r ) , we must also have thatthe probability that the point ˆ ◊ = 1 1 ≠  ( ◊ + › ) is in K (and is therefore not rejected) is greater than810 4 .
This implies that the number of iterations until our algorithm returns a point ˆ ◊ is less thank > 0 with probability at least 1 ≠ 2 ≠ k , and the expected number of iterations is at most 2 (provedin Corollary B.4).
Since each iteration requires one random sample ◊ from the distribution µ , and one call to a membership oracle for K (to determine if 1 1 ≠  Z œ K ), the number of sampling oracle and membershiporacle calls required by Algorithm 1 is O (1) with very high probability.
Therefore, with high probability, Algorithm 1 returns a point ˆ ◊ from a distribution with inﬁnity-distance at most Á from ﬁ after
O (1) calls to the sampling and membership oracles.
Since Algorithm 1 succeeds with probability 1 ≠ 2 ≠ k after k iterations, after · max = 5 d log( Rr ) +5 LR + Á iterations Algorithm 1 will have succeeded with probability roughly 1 ≠ Á ( Rr ) ≠ 5 d e ≠ 5 LR .
In the very unlikely event that Algorithm 1 still has not succeeded after · max iterations, Algorithm1 simply outputs a point sampled from the uniform distribution on the ball B (0 , r ) of radius rcontained in K .
The probability mass of the target distribution ﬁ inside this ball is at least as large as ( Rr ) ≠ d e ≠ LR ; thus, since f is L -Lipschitz, we show in Corollary B.4 that outputing a sample fromthe uniform distribution on this ball with probability Á ( Rr ) ≠ 5 d e ≠ 5 LR does not change the Œ -distanceerror of the sample returned by the algorithm by more than Á .3.
Completing the proof of Theorem 2.
Proof: By Theorem 1, the output of Algorithm 1 has inﬁnity-distance error bounded by Á as long as the input samples have TV distance error bounded by ” Æ O [Equation]
[Equation]
and, with high probability, Algorithm 1 requires O (1) such independent samples.
To generate a sample from ﬁ with TV error O ( ” ) when K = { ◊ œ R d : A ◊ Æ b } is a polytope deﬁned by m inequalities, we use the Dikin Walk Markov chain of [34].
This Markov chain requires an initial point from some distribution µ 0 which is w -warm with respect to the stationary distribution ﬁ , that is, sup z œ K µ
( z ) ﬁ ( z ) Æ w .
To obtain a warm start, we let µ 0 be the uniform distribution on the ball with radius r contained in K and sample from µ 0 .
Since f is L -Lipschitz, and K is contained in a ball of radius R , µ 0 is w -warm with
[Equation]
From [34], we have that from this w -warm start the Dikin Walk Markov chain requires at most
O (( m 2 d 4 + m 2 d 2 L 2 R 2 ) log( w ” )) steps to generate a sample with TV distance at most ” from ﬁ ,where each step makes one function evaluation and O ( md Ê ≠ 1 ) arithmetic operations.
Plugging inthe above values of ” , w the number of Markov chain steps is T
= O (( m 2 d 3 + m 2 dL 2 R 2 )
◊ [ LR +d log( Rd + LRdr Á )]) to generate each independent sample with the required TV error O ( ” ) .
Sincethe number of independent samples required as input for Algorithm 1 is O (1) w.h.p., the numberof arithmetic operations for Algorithm 1 to output a point with at most Á inﬁnity-distance error is
O ( T ◊ md Ê ≠ 1 ) .
Finally, we note that in the more general setting where K is a convex body with membership oracle (but not necessarily) a polytope, we can instead use, for instance, the hit-and-run Markov chain of [29] to generate samples from ﬁ with TV error O ( ” ) in a number of membership and function evaluation oracle calls that is polynomial in d and poly-logarithmic in 1 ” , R, r .
We can then plug this sample into our Algorithm 1 to obtain a sample from ﬁ with inﬁnity-distance error O ( Á ) in a number of oracle calls that is (poly)-logarithmic in 1 Á , 1 r and polynomial on d, L, R .
(see Remark 2.3).
Conclusion, Limitations, and Future Work
To the best of our knowledge, this is the ﬁrst work that presents an algorithm for sampling from logconcave distributions on convex bodies that comes with inﬁnity-distance bounds and whose runningtime depends logarithmically on 1 / Á .
Towards this, the main technical contribution is Algorithm1 (and Theorem 2.2) which achieves this improved dependence on Á by taking as input continuoussamples from a convex body with TV bounds and converting them to samples with inﬁnity-distancebounds.
On the other hand, our bounds are polynomial in LR , yet there are algorithms for sampling fromlogconcave distributions
ﬁ Ã e ≠ f on a convex body in the total variation distance that are polylogarithmic in R and do not assume f to be Lipschitz [29].
Thus, the main open problem thatremains is whether one can also obtain running time bounds for sampling in the inﬁnity-distancewhich are poly-logarithmic in R and do not require f to be Lipschitz.
Our main result also has direct applications to differentially private optimization (Corollaries 2.4and 2.5).
Differential privacy is a notion which has been embraced in many technologies in societalcontexts where privacy of individuals is a concern.
Hence, we see our work to have a potential ofpositive societal impact and do not foresee any potential negative societal impacts.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for informationon how to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .
You are strongly encouraged to include a justiﬁcation to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[N/A]
• Did you include the license to the code and datasets?
[N/A]• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] (SeeSection 4)
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes] (See Section 3.2,
Appendix B and C)
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[N/A]
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[N/A]
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[N/A]
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[N/A]
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[N/A]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data
you’re using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁ-
able information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]