An Error Analysis of Deep Density-Ratio Estimation with Bregman Divergence
Anonymous Author(s)
AfﬁliationAddress email
Abstract
We establish non-asymptotic error bounds for a nonparametric density-ratio estimator using deep neural networks with the Bregman divergence.
We also show thatthe deep density-ratio estimator can mitigate the curse of dimensionality when thedata is supported on an approximate low-dimensional manifold.
Our error boundsare optimal in the minimax sense and the pre-factors in our error bounds dependon the dimensionality of the data polynomially.
We apply our results to investigatethe convergence properties of the telescoping density-ratio estimator (Rhodes et al.
and provide sufﬁcient conditions under which it has a smaller upper errorbound than a single-ratio estimator
Introduction
Density-ratio estimation is of key importance in various statistical and machine learning problems(Sugiyama et al., 2012b; Kato & Teshima, 2021).
There is a vast literature on density-ratio estimationdue to its wide range of applications, such as discriminative analysis (Silverman, 1978; Cox & Ferry,1991), covariate shift adaptation (Sugiyama et al., 2008; Tsuboi et al., 2009), two-sample testing(Qin, 1998; Sugiyama et al., 2011), energy-based modelling (Gutmann & Hyvärinen, 2012; Ceylan &
Gutmann, 2018), generative learning (Goodfellow et al., 2014;
Nowozin et al., 2016), and mutualinformation estimation (Moustakides & Basioti, 2019; Rhodes et al., 2020), among others.
Let Z q and Z p ∈ Z
=
[0 , 1] d be two random vectors with probability density functions q ∗ and p ∗ ,respectively.
Given independent and identically distributed (i.i.d) samples { Z q,i } n q i =1 from q ∗ and{ Z p,j } n p j =1 from p ∗ , a basic problem is to estimate the density ratio
R ∗ ( z )
[Equation]
[Equation]
A naive estimator of R ∗ is ˆ q/ ˆ p , where ˆ q and ˆ p are the density estimators of q ∗ and p ∗ , respectively.
However, such an estimator can be highly unstable.
Moreover, density estimation itself is a difﬁcult problem, especially in the high-dimensional settings.
For example, kernel density estimators(Rosenblatt, 1956; Parzen, 1962) works well when d ≤ 3 , but deteriorate dramatically as d increases.
To avoid density estimation, various methods have been proposed to estimate the density ratio R ∗directly, including the density matching approach (Sugiyama et al., 2008; Tsuboi et al., 2009; Yamada& Sugiyama, 2009; Nguyen et al., 2010; Yamada et al., 2010), the moment matching approach (Qin,1998; Gretton et al., 2009; Kanamori et al., 2012b), the density-ratio ﬁtting approach (Kanamoriet al., 2009, 2012a), and the uniﬁed density-ratio matching approach under Bregman divergenceframework (Sugiyama et al., 2012a).
Impressive empirical successes of using deep neural networksin density-ratio estimation have been reported in some recent works (Moustakides & Basioti, 2019;
Rhodes et al., 2020).
Moreover, Kato & Teshima (2021) studied the convergence properties of deepdensity-ratio estimation under a modiﬁed Bregman divergence criterion.
In this paper, we study deep density-ratio estimators with the Bregman divergence as the criterion.
Weapply our results to construct an estimator for statistical inference for the Kullback-Liebler divergence.
We also study the theoretical properties of the telescoping density-ratio estimator (Rhodes et al.,2020) based on our results.
Our contributions are as follows:
We establish non-asymptotic error bounds for the density-ratio estimator using deep neuralnetworks under the Bregman divergence (BD, Bregman, 1967), and provide a neural networkarchitecture for the estimator to achieve minimax optimal rate O p (cid:0) n − 2 β/
( d +2 β ) (cid:1) , wheren = min { n q , n p } and β is a smoothness parameter of the logarithmic density-ratio function;see Subsection 3.2 for details;2.
We show that deep density-ratio estimator with the Bregman divergence criterion is ableto mitigate the curse of dimensionality when the data is supported on an approximatelow-dimensional manifold; see Subsection 3.3;3.
We apply our results to study the convergence properties of the telescoping density-ratio
estimator (Rhodes et al., 2020) and demonstrate its advantages over single-ratio estimatorsunder certain conditions.
Notation.
Let n = min { n q , n p } be the smaller sample size between the two samples { Z q,i } n q i =
1and { Z p,j } n p j =1 .
In addition, (cid:107) · (cid:107) ∞ denotes the sup-norm on some speciﬁc domain, and C, C 0 aregeneric constants that may vary from place to place.
For any measurable function f , we denote(cid:107)
f (cid:107) max := max {(cid:107) f
(cid:107) p , (cid:107) f
(cid:107) q } and (cid:107) f (cid:107) n p ,n q = max {(cid:107) f (cid:107) p,n p , (cid:107) f (cid:107) q,n q } , where (cid:107) f (cid:107)
I
=
E I ∗ f 2 ( Z )and (cid:107) f (cid:107)
I,n I = E n I f 2 ( Z ) =
(1 /n I ) (cid:80) n I t =1 f
( Z I,t ) , I = p, q .
Density-ratio estimation
In this section, we ﬁrst present the density-ratio estimation problem using the Bregman divergence(BD, Bregman, 1967) and then describe the structure of the deep neural networks to be used indensity-ratio estimation.
Let ψ : R → R be a ﬁrst-order continuously differentiable and strictly convex function.
Deﬁne
[Equation]
where ψ (cid:48) is the derivative of ψ .
Then, the convexity of ψ implies that ∆ ψ ( x, y ) ≥ 0 and the equalityholds if and only if x = y .
It follows that E p ∗ ∆ ψ ( R ∗ ( Z ) , R ( Z )) ≥ 0 and the equality holds if andonly if R = R ∗ .
Therefore, the target density-ratio R ∗ =
q ∗ /p ∗ can be characterized as a minimizer:
[Equation]
We verify in the appendix that
[Equation]
Since the last term on the right side in (1) E p ∗ [ ψ ( R ∗ ( Z ))] is independent of R , we have
[Equation]
and
[Table]
[Equation]
Hence, for any measurable function R : Z → R , the BD score induced by ψ for estimating the targetdensity-ratio R ∗ =
q ∗ /p ∗ is
[Equation]
where ψ (cid:48) is the derivative of ψ (Sugiyama et al., 2012a,b).
Then, R ∗ is the minimizer of B ψ ( R ) overall nonnegative measurable functions.
Because a density ratio is always nonnegative, a nonnegative constraint needs to be considered whendeﬁning the density ratio as a minimizer, as in (2).
To avoid the non-negative constraint of the density ratio, we ﬁrst consider the log-density ratio D ∗ := log R ∗ .
Then the nonnegativity constraint is nolonger needed and by (2), we have
[Equation]
In practice, the estimation of R ∗ can be based on an empirical version of B ψ when random samplesfrom p ∗ and q ∗ are available.
Suppose we have samples { Z q,i } n q i =1 i.i.d.
q ∗ and { Z p,j } n p j =1 i.i.d.
p ∗ .
We estimate D ∗ by
[Equation]
where F n is a class of neural network functions and (cid:98) B ψ ( e D ) is an empirical version of B ψ ( e D )deﬁned in (3), which can be written as
[Equation]
where
[Equation]
The density-ratio estimator is (cid:98) R = exp( (cid:98) D ) .
We take the function class F n to be F M, D , W , U , S , a class of ReLU activated feedforward neuralnetworks (FNNs)
f θ : R d → R with parameter θ , depth D , width W , size S , number of neurons
U .
We require that (cid:107) f θ (cid:107) ∞ ≤ M for some 0 ≤ M ≤ ∞ .
There are D hidden layers and ( D + 1)layers in total.
The width W is the maximum width of the hidden layers; the number of neurons U isdeﬁned as the number of neurons of f θ ; the size S is the total number of parameters in the network.
Note that D , W , U , S may depend on n , but we suppress the dependence for notational simplicity.
We write F M, D , W , U , S as F FNN for brevity.
Theoretical results
In this section, we ﬁrst study the error bounds for the deep logarithmic density-ratio estimator.
Thebounds for the density-ratio estimator follows directly based on the properties of the exponentialfunction.
We also show that deep density-ratio estimator can mitigate the curse of dimensionalitywhen data is supported on an approximate low-dimensional manifold.
3.1 General error bounds
To state our assumptions and results, we need the deﬁnitions of µ -smoothness, σ -strong convexityand pseudo dimension.
A function f : R → R is said to be µ -smooth over a set A ⊆ R if it is differentiable over A and itsﬁrst-order derivative f (cid:48) satisﬁes
[Equation]
(6) where 0 ≤ µ < ∞ .
The constant µ is called the smoothness parameter.
A function f : R → R is called σ -strongly convex if the domain dom ( f ) of f is convex and for anyx, y ∈ dom ( f ) and λ ∈ [0 , 1] , f satisﬁes
[Equation]
(7) where 0 ≤ σ < ∞ .
The constant σ is called the strong convexity (SC) parameter.
For a function class F , its pseudo dimension denoted by Pdim ( F ) , is the largest integer B satisfyingthat there exists ( x 1 , x 2 , . . .
, x B , y 1 , y 2 , . . .
, y B ) ∈
Z B × R B such that for any ( r 1 , r 2 , . . .
[Table]
[Table]
KL c log c − c [ a, b ] (0 ≤ a ≤ b )
[Table]
[Table]
Remark 1.
For any measurable function class F , by the deﬁnition of VC dimension,
VCdim ( F ) ≤ Pdim ( F ) .
If F is the class of functions generated by ReLU FNNs, it follows from
Theorem 14.1 of Anthony & Bartlett (1999) that Pdim ( F ) ≤
VCdim ( F ) .
Hence, for the functionclass F generated by ReLU FNNs, Pdim ( F ) =
VCdim ( F ) .
We make the following assumptions.
Assumption 1.
The function ψ is µ -smooth & σ -strongly convex, that is, it satisﬁes (6) and (7).
Some commonly-used ψ ’s satisfy Assumption 1; see Table 1 for some examples.
Assumption 2.
There exists a constant 0 < M < ∞ such that (cid:107) D ∗ (cid:107) ∞ ≤ M, (cid:107) D (cid:107) ∞
≤ M forevery D ∈ F FNN .
Assumption 2 assumes that the target density ratio is bounded.
Such an assumption is often made innonparametric statistics for avoiding technical difﬁculties associated with dealing with unboundedfunctions.
We will partially relax this assumption below.
The ﬁnite M in Assumption 2 canbe relaxed to M =
O (log log n ) at a small price of an additional logarithm term in the errorbounds.
The boundedness of a network can be achieved by clipping operation.
For example, let
T M ( t ) = − MI { t < − M } + tI {− M ≤ t ≤ M } + MI { t > M } be the truncation function takingvalues in [ − M, M ] , then T ( t )
=
σ ( t ) − σ ( σ ( t ) − M ) −{ σ ( − t ) − σ ( σ ( − t ) − M ) } can be computedby a ReLU network with depth 2 and width 4 .
Hence, through network concatenation, we canconstruct some bounded ReLU FNNs and such a boundedness assumption can be satisﬁed.
Deﬁne the best in class approximation of D ∗ in F FNN as D NN ∈ arg min D ∈F FNN (cid:107) D − D ∗ (cid:107) max .
Denote
[Equation]
Theorem 1.
Suppose Assumptions 1-2 are satisﬁed.
When n ≥ Pdim ( F FNN ) , there exists a 125 constant C depending on ( µ, σ, M ) such that for any γ > 0 , with probability at least 1 − exp( − γ ) ,
[Equation]
and
[Equation]
We have the following corollary for the expected error.
Corollary 1.
Under the conditions of Theorem 1, there exists a constant C 0 depending only on( µ, σ, M ) , such that
[Equation]
and
[Equation]
The above results are obtained under the boundedness Assumption 2.
While such an assumption isoften made in the error analysis of nonparametric procedures, it is somewhat restrictive in densityratio estimation problems.
For example, this assumption may not be satisﬁed in the presence ofthe density-chasm problem, i.e., the gap between two densities is large (Rhodes et al., 2020).
Weestablish an error bound result with the following partially relaxed assumption.
Assumption 3.
There exists a constant 0 < M < ∞ such that D ∗ ( z ) ≥ − M for every z ∈ Zand (cid:107) D (cid:107) ∞ ≤ M for every D ∈ F FNN .
This assumption does not require the target log-density ratio D ∗ to be bounded above.
Denotetruncated versions of D ∗ and R ∗ by
[Equation]
where 0 < M < ∞ and 1 {·} is the indicator function.
We establish a non-asymptotic error boundinvolving the truncation error.
Theorem 2.
Suppose Assumptions 1 and 3 hold.
When n > Pdim ( F FNN ) , there exists twoconstants C depending only on ( µ, σ, M ) and C 0 depending only on ( µ, σ ) , such that
[Equation]
where ξ n is deﬁned in (8).
The term (cid:107) R ∗ − R ∗ M (cid:107) 2 p is the truncation error for an unbounded R ∗ and the unboundedness alsoleads to the term ξ n =
[ Pdim ( F FNN )(log n ) /n ] 1 / 2 in the error bound, which is greater than ξ 2 n inthe bounded case.
However, because no boundedness assumption is needed in this theorem, we canapply it to study the convergence properties of the telescoping density-ratio estimator of Rhodes et al.(2020) in Section 4 below.
3.2 Non-asymptotic
error bounds
By Corollary 1, it sufﬁces to bound the estimation error Pdim ( F FNN ) log n/n and the approximation error (cid:107) D NN − D ∗ (cid:107) 2max .
It follows from Theorem 6 in Bartlett et al.
(2019) that, for
F FNN = F M, D , W , U , S , there exists a universal constant C 2 such that Pdim ( F FNN ) ≤ C 2 SD log S .
To control the approximation error (cid:107) D NN − D ∗ (cid:107) 2max , we assume that D ∗ belongs to the Hölderclass H β ([0 , 1] d , M ) with β
= k + a where k ∈ N + and a ∈ (0 , 1] , where N + is the set of positiveintegers.
Deﬁnition 1 (Hölder class) .
A Hölder class H β ([0 , 1] d , M ) with β = k
+ a where k ∈ N + anda ∈
(0 , 1] consists of function f : [0 , 1] d → R satisfying
[Equation]
where (cid:107) α (cid:107)
=
(cid:80) di
=1 α i and ∂
α = ∂
α 1 ∂ α 2 · · · ∂ α d for α
= ( α 1 , α 2 , . . .
, α d ) ∈ N + d .
We use Theorem 3.3 of Jiao et al.
(2021) to control the approximation error (cid:107) D NN − D ∗ (cid:107) 2max .
Forconvenience, we include this result in the following lemma.
We specify the width W and depth D as follows.
For any K, L ∈ N + ,
[Equation]
+ 1)
2 d (cid:98) β
(cid:99)
+1 L (cid:100) log 2 (8 L ) (cid:101) , D =
( (cid:98) β (cid:99) + 1) 2 K (cid:100) log 2 (8 K ) (cid:101) , (9) where (cid:100) a (cid:101) is the smallest integer no less than a .
Lemma 1 (Approximation error) .
Assume f ∈ H β
([0 , 1] d , M ) with β = k
+ a where k ∈ N +and a ∈ (0 , 1] .
Then there exists a function φ 0 implemented by a ReLU network with width W anddepth D speciﬁed in (9) such that
[Equation]
where C β = ( (cid:98) β (cid:99) + 1) 2 d (cid:98) β (cid:99) +( β ∨ 1) / 2 , H B,δ = ∪ di =1 { x = [ x 1 , . . .
, x d ] : x i ∈∪ B − 1 b =1 ( b/B − δ, b/B ) } for B = (cid:100) ( KL ) 2 /d (cid:101) , δ ∈ (0 , 1 / (3 B )] and a ∨ b = max( a, b ) .
Furthermore, if W = 38( (cid:98) β (cid:99) + 1) 2 d (cid:98) β (cid:99) +1 3 d L (cid:100) log 2 (8 L ) (cid:101) , D =
( (cid:98) β (cid:99) + 1) 2 K (cid:100) log 2 (8 K ) (cid:101) + 2 d,then
[Equation]
In this uniform approximation result, the width W is required to depend on d exponentially.
The following theorem gives an error bound for (cid:98) D .
Theorem 3 (Non-asymptotic error bound for (cid:98) D ) .
Suppose that Assumptions 1-2 are satisﬁed,
D ∗ ∈ H β
[0 , 1] d , M ) with β = k
+ a where k ∈ N + and a ∈ (0 , 1] , and F FNN is the function classof ReLU DNNs with width W and depth D speciﬁed in (9).
Then, for M ≥ 1 and n ≥ Pdim ( F FNN ) ,we have
[Equation]
where C 1 = ( (cid:98) β (cid:99) + 1) 4 d 2 (cid:98) β (cid:99) +( β ∨ 1) and the constant C depends only on ( µ, σ, M ) .
Furthermore, if
[Equation]
then
[Equation]
where the constant C 0 depends only on ( µ, σ, M ) .
The convergence rate in (10) is optimal.
This can be seen by considering a density estimation problemwith i.i.d observations { Z (1) q,i } m q i =1 from an underlying unknown density q 1 on [0 , 1] d .
To estimateq 1 , we sample referencing observations { Z (1) p,j }
m p j =1 with m p ≥ m q , from a uniform distribution
Unif ([0 , 1] d ) whose density p 1 ≡ 1 .
Thus, estimating the density ratio q 1 /p 1 is equivalent toestimating q 1 .
According to (4), we obtain the estimator ˆ q 1 of q 1 .
If log q 1 ∈ H β ([0 , 1] d , M ) whereβ =
k
+ a with k ∈ N + and a ∈ (0 , 1] , a neural estimator based on the network structure speciﬁedin Theorem 3 satisﬁes
[Equation]
Tsybakov (2008) showed that for a density belonging to the Hölder function class, the optimalminimax rate of the density estimation is O p (cid:16)
m − 2 β/
( d +2 β ) q (cid:17) .
Hence, our estimator achieves theoptimal minimax rate.
In addition, the existing error bounds usually contain a prefactor depending on the dimension dexponentially, e.g. 2 d (Devroye & Lugosi, 1996).
Such a prefactor can be very large even for amoderately large d , which severely degrades the quality of an error bound.
The prefactors in ourresults depend on d only polynomially and are much smaller than those in the existing bounds.
Under Assumption 2, to derive a nonasymptotic error bound for the log-density ratio estimator (cid:98) R , wenote that
[Equation]
Thus a bound for (cid:98) R follows directly from a bound for (cid:98) D.
Remark 2.
Appendix A.2 contains some examples of p ∗ and q ∗ such that D ∗ = log( q ∗ /p ∗ ) ∈
H β ([0 , 1] d , M ) .
Remark 3.
The hypercube [0 , 1] d assumption for the density ratio is made for technicalconvenience.
With an unbounded support, we can bound (cid:107) D NN − D ∗ (cid:107) max using a truncationtechnique under some suitable additional assumptions, at a small price of an additional logarithm term in the error bound.
Speciﬁcally, suppose the pdfs are supported on R d .
In additionto Assumptions 1-2 and the Hölder class assumption in Theorem 3, we need to further assumethat max { E p ∗ I ( (cid:107) Z (cid:107) ∞ ≥ log n ) , E
[Equation]
I ( (cid:107) Z (cid:107) ∞ ≥ log n ) } ≤ n − 2 β d +2 β .
For I = p or q , and any
D ∈ F FNN , where F FNN is the function class of ReLU FNNs with width W and depth D speciﬁed by
[Equation]
we have
[Equation]
Compared with the upper bound of the approximation error in Theorem 3, when the pdfs are supported
R d (unbounded case) we can derive a similar approximation error upper bound with an additional factor ( n )
3.3 Circumventing the curse of dimensionality
In many modern statistical and machine learning tasks, such as image processing and text analysis,the dimensionality d of the data can be high, which results in a very slow convergence rate even witha large sample size.
This is known as the curse of dimensionality.
Nonetheless, the data in variousapplications has been demonstrated to be supported or approximately supported in some subspacesor subsets with low intrinsic dimensionality (Nakada & Imaizumi, 2020).
For regression problems,
Nakada & Imaizumi (2020) have shown that DNNs can adaptively estimate the regression functionthrough the low-dimensional structure of the data, and the resulting convergence rates no longerdepend on the nominal high dimensionality d of the data, but on its low intrinsic dimension.
Motivated by these advancements, we assume that the data is concentrated on an approximate compact
Riemannian submanifold M with the Riemannian dimension d M (cid:28) d .
Assumption 4.
The target log-density ratio D ∗ ∈ H β ([0 , 1] d , M ) with β = k
+ a wherek ∈ N + and a ∈ (0 , 1] , and
the data from the densities p ∗ , q ∗ are concentrated on a set M ρ ⊆ [0 , 1] ddeﬁned as
[Equation]
where M is a compact d M -dimensional Riemannian submanifold and ρ ∈ (0 , 1) .
Theorem 4.
Suppose Assumptions 1, 2 and 4 hold.
Suppose that D ∗ ∈
H
β
([0 , 1] d , M ) withβ =
k
+ a , k ∈ N + and a ∈ (0 , 1] .
If F FNN is the function class of ReLU FNNs with width and depth
[Equation] [Equation]
K
[Equation]
log
[Equation]
8 K ) (cid:101) , where K, L ∈ N + and d δ = O (cid:0) d M log( d/δ ) /δ
2 (cid:1) (cid:28) d , then when M ≥ 1 , n > Pdim ( F FNN ) and
[Equation]
we have
[Equation]
where the constant C only depends on ( µ, σ, M ) , C
= ( (cid:98) β (cid:99) + 1) 4 (2 d ) 2 β d 3 β
+( β
[Equation]
1) δ , and ξ n isdeﬁned in (8).
By Theorem 4, if we set W = 114( (cid:98)
[Equation]
8 n ζ δ (cid:1)(cid:7) , withζ δ
=
d δ / (2( d δ + 2 β )) , then
[Equation]
(12) where the constant
Conly depends on ( µ, σ, M ) and
C=
[Equation]
β
[Equation]
+
[Equation]
+3 δ , (2 d ) 2 β d 3 β
+( β ∨ 1) δ } .
The convergence rate n − 2 β/ ( d δ +2 β ) in (12) only depends on d δ (cid:28) d , instead of the ambient dimension d .
Therefore, Theorem 4 shows that alow-dimensional Riemannian manifold support assumption can alleviate the curse of dimensionality.
Error analysis of the telescoping density-ratio estimator
When the difference or the ‘gap’ between two densities is large, a single-ratio estimation methodmay perform poorly.
This is referred to as the the density-chasm problem (Rhodes et al., 2020).
Toalleviate this problem, Rhodes et al.
(2020) proposed an approach called Telescoping density-Ratio
Estimation (TRE).
This approach ﬁrst gradually transports samples from q ∗ to samples from p ∗ ,creating a chain of intermediate datasets, then estimates the density ratio between consecutive datasetsalong this chain.
The chained ratios are combined via a telescoping product which yields an estimateof the original density ratio.
The experiments conducted by Rhodes et al.
(2020) demonstrate that
TRE can yield substantial improvements over existing single-ratio methods for mutual informationestimation, representation learning and energy-based modelling.
We now provide a theoretical analysis of TRE, which partially explains why TRE performs well.
Fornotational simplicity, suppose n p = n q ≡ n below.
For k = 0 , 1 , . . .
, K , Rhodes et al.
(2020) constructed a chain of intermediate samples connectingq ∗ and p ∗ by setting Z k,i = (1 − α 2 k ) 1 / 2 Z q,i + α k Z p,i , i = 1 , . .
. , n, where
=
α 0 < α 1 < · · · <α K − 1 < α K = 1 , and used these samples to build a TRE.
To simplify the analysis, we use a slightly different chain of intermediate samples as follows.
Fork = 0 , 1 , . . .
, K , let Z k,i =
(1 − δ k,i )
Z q,i + δ k,i Z p,i , i = 1 , . . . , n, where δ k,i , i = 1 , . . . , n, arei.i.d.
Bernoulli random variables with success probability α k .
Let q k be the density of the synthetic data Z k,i constructed this way.
We have q k ( z ) =
(1 − α k ) q ∗ ( z )+α k p ∗ ( z ) , k = 1 , . .
, K − 1 .
Therefore, the distribution of the samples from q k in the chain is asimple mixture of q ∗ and p ∗ with the mixing proportions 1 − α k and α k , instead of a more complexconvolution of two densities using the construction of Rhodes et al.
As α k changes fromα 0 = 0 to α K = 1 over a grid { α 0 , α 1 , . .
. , α K } ⊂ [0 , 1] , the distributions of the samples in thechain move gradually from q ∗ to p ∗ .
Let q 0 = q ∗ and q K ≡ p ∗ .
Then,
[Equation]
(13) where R ∗ i ( z )
=
q i ( z ) /q
i +1 ( z ) .
For k = 0 , 1 , . . .
, K − 1 , applying the neural density-ratio estimatorwith { Z k,j } n k j =1 and { Z k +1 ,j } n k +1 j =1 yields an estimator (cid:98) R i of R ∗ i .
Then the telescoping density ratioestimator of R ∗ is (cid:81) K − 1 i =
(cid:98) R
i
We consider the log-density ratio.
Let (cid:98)
D
k be the neural estimator of D ∗ k ≡ log( q k /q k +1 ) .
Based on(13), the telescoping estimator of the log-density ratio D ∗ ≡ log R ∗ is
[Equation]
In what follows, we show that under certain conditions, the telescoping estimator has an improvedasymptotic error bound.
The intuition is as follows: when q k /q k +1 is bounded or q k ( z ) /q
k +1 ( z ) (cid:28)q ∗ ( z ) /p ∗ ( z ) for z ∈ Z , where q k and q k +1 are the densities of the synthetic data { Z k,j } nj =1 and{ Z k +1 ,j } nj =1 , respectively, the truncation error for q k /q k +1 vanishes or is far less than that for q ∗ /p ∗ .
This can help the telescoping density-ratio estimator perform better than a single-ratio estimator.
Assume that q ∗ ≥ c 1 and c 1 ≤ p ∗ ≤ c 2 , where 0 < c 1 , c 2 < ∞ are two constants.
Thus,
D ∗ = log( q ∗ /p ∗ ) ≥ log( c 1 /c 2 ) .
Therefore, Assumption 3 is satisﬁed.
For any ﬁnite set A ⊂ R ,max A denotes the maximal value in A .
Let M = log( c 2 /c 1 ) and M 0 be a constant satisfying
[Equation]
(15) where
[Equation]
Based on Theorem 2, we can establish an asymptotic error bound for the telescoping estimator (cid:98) D TREdeﬁned in (14), with
[Equation]
where
[Equation]
where L 1 and L 2 are deﬁned in (5) and F 0 FNN consists of DNNs D with (cid:107) D (cid:107) ∞ ≤ M 0 .
Todemonstrate the advantages of the telescoping estimator, we also consider the single-ratio estimator(SRE), (cid:98) D SRE ∈ arg min D ∈F 0 FNN ˆ B ψ ( e D ) .
Proposition 1.
Assume that q ∗ ≥ c 1 , c 1 ≤ p ∗ ≤ c 2 , where the constants 0 < c 1 ≤ c 2 < ∞ , and the samples { Z q,i } ni =1 from q ∗ and { Z p,j } nj =1 from p ∗ are independent.
Then, there exists a constant C 0 ( µ, σ, c 1 ) depending only on ( µ, σ, c 1 ) such that for
[Equation]
we have
[Equation]
where (cid:107) f
(cid:107) 2 = (cid:2)(cid:82)
Z f 2 ( z ) dz (cid:3) 1 / 2 for any square integrable function f .
Proposition 1 shows that for a given sequence 0 =
α 0 < α 1 < · · · < α K − 1 < α K = 1 and atruncation level M 0 , the upper bound for the asymptotic L 2 -error of (cid:98) D TRE is reduced by a factor(1 − α K − 1 ) with 0 < 1 − α K − 1 < 1 .
This upper bound can be far less than that of (cid:98) D SRE whenα K − 1 is close to 1.
Therefore, TRE can improve the asymptotic error bound over the bound for thesingle-ratio method.
It is important to note that there is a tradeoff between the value of α K − 1 and the truncation level
M 0 dictated by (15).
For instance, with α 1 ≤ · · · ≤ α K − 2 ﬁxed, the closer α K − 1 is to 1, inview of (15), the larger M 0 is.
Larger α K − 1 sharpens the pre-factor (1 − α K − 1 ) and larger M 0 alsoimproves (cid:107) R ∗ − R ∗ M
(cid:107) p , but deteriorates the pre-factor e M 0 .
Proposition 1 is generally not applicable to the original chain of TRE.
The difﬁculty is due tothe possibly intensive oscillation of density ratios caused by the convolution form for the densityof the sum of two random variables.
We illustrate this by a toy example: suppose Z q , Z p arei.i.d.
uniform random variables on [0 , 1] .
For any t ∈ (0 , 1 / 2] , (1 − t )
Z q + tZ p has densityq ∗ t ( z ) =
z t (1 − t )
I { 0 ≤ z ≤ t } + 11 − t
I { t < z ≤ 1 − t } + 1 − z t (1 − t ) I { 1 − t <
z ≤ 1 } .
In this case, q ∗ /q ∗ tis unbounded and oscillates sharply when z is close to 0 or 1.
This makes it hard to estimate q ∗ /q
∗ t .
However, the chain we used does not have this problem, which may be a good choice in practice.
Additionally, we conduct simulation studies to evaluate the performance of our proposed mixingchain and the original convolution chain; see Table 2 for the results.
The simulation settings aregiven in Appendix A.4.
Table 2 shows that, for the models considered in the simulation studies, theproposed mixing chain performs comparably or better compared with the original convolution chain.
[Table]
Related work: comparison with the NN-BD estimator
There has been much work on the error analysis of nonparametric density-ratio estimation (Nguyenet al., 2010; Sugiyama et al., 2008; Kanamori et al., 2012a; Yamada et al., 2013).
These results showthat when the targeted density-ratio belongs to certain function space H , such as RKHS, and thusno approximation error is incurred, their estimators achieve certain nonparametric convergence ratedecided by the complexity of H .
Compared to these works, our results consider the approximationerror using neural network functions and still achieves the minimax optimality under some mildconditions.
Our work is most related to the paper by Kato & Teshima (2021), who proposed a non-negative
Bregman divergence (NN-BD) method to tackle the possible over-ﬁtting problem due to the unboundedness of certain Bregman divergences.
We compare our theoretical results with those for the NN-BD estimator of Kato & Teshima (2021).
Using the notation in this paper, we restate two conditionsrequired in Kato & Teshima (2021):(a) Let F R FNN be a class of FNNs with output taking values in [ e − M , e M ] for some ﬁnite M > 0 .
The target density-ratio R ∗ ∈ F R FNN .
Moreover, for any function in F R FNN , its Frobenius norm ofthe parameter matrix W j in the j th layer is bounded by B j ≥ 0 and the activation functions are1 -Lipschitz positive-homogeneous.(b)
The function ψ ( · ) is σ -strongly convex.
Let (cid:96) 1 ( t ) =
ψ ∗ ( t ) t − ψ ( t ) + A, (cid:96) 2 ( t ) = − ˜ ψ ( t ) , t ∈[ e − M , e M ] , where ψ ∗ ( t ) =
C nn { ψ (cid:48) ( t ) t − ψ ( t ) }
+ ˜ ψ ( t ) .
Here ˜ ψ ( t ) is a function bounded above, C nnand A are user-selected constants.
Suppose (cid:96) 1 ( · ) and (cid:96) 2 ( · ) are Lipschitz functions on [ e − M , e M ] .
Under these two conditions, Kato & Teshima (2021) rewrote the BD in (3) as
[Equation]
E
[Equation]
and proposed the density-ratio estimator (cid:98) R KT deﬁned as
[Equation]
where [ a ] + = max(0 , a ) for any a ∈ R .
They showed that
[Equation]
for any 0 ≤ a ≤ 2 .
According to Theorem 1, we have the following corollary for our density-ratio estimator (cid:98) R = exp( (cid:98) D ) .
Corollary 2.
Under Assumption 1 , when n ≥ Pdim ( F R FNN ) , there exists a constant C dependingonly on ( µ, σ, M ) such that, for any γ ≥ 0 , with probability at least 1 − exp( − γ ) ,
[Equation]
Corollary 2 implies that (cid:107) (cid:98) R − R ∗ (cid:107) p =
O p (cid:0)(cid:112) log n/n (cid:1) , when the true density-ratio R ∗ ∈ F FNN .
Thisconvergence rate is slightly faster than the rate for (cid:98) R KT given in (17).
Moreover, the boundednessassumption for the weights of the neural network functions, as imposed by Kato & Teshima (2021),is not needed.
Corollary 2 also shows that, if the target ratio is assumed to belong to the optimizationspace (or hypothesis space), i.e., R ∗ ∈ F R FNN without approximation error, then the convergence ratedoes not depend on the dimension of the data.
In other words, the estimation of R ∗ does not sufferfrom the curse of dimensionality.
However, this is probably not realistic.
Therefore, it is important toconsider the approximation error due to the fact that R ∗ (cid:54)∈ F R FNN in applications.
Conclusions
In this paper, we have established the non-asymptotic error bounds for the deep density-ratio estimatorusing the Bregman divergence criterion.
Under reasonable conditions, we have shown that the deepdensity-ratio estimator achieves the optimal minimax convergence rate.
When the data is supportedon an approximate low-dimensional manifold, we have shown that the neural estimator can mitigatethe curse of dimensionality.
We have also analyzed the convergence properties of the telescopingdensity ratio estimator (Rhodes et al., 2020) and provided sufﬁcient conditions under which it has alower error bound than a single-ratio estimator.
A limitation of this work is that certain boundedness assumptions on the target density ratio suchas Assumption 2 or 3 is needed.
Also, when the boundedness assumption is partially relaxed as in
Assumption 3, the error bound in Theorem 2 is not as sharp as that with the boundedness assumptionin Theorem 1.
It would be interesting to further relax or remove such assumptions.
It would also beuseful to improve the error bound in Theorem 2 if possible.
These are interesting and challengingproblems that deserve further study in the future.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’scontributions and scope?
[Yes](b) Did you describe the limitations of your work?
[Yes] See Section 6.
(c) Did you discuss any potential negative societal impacts of your work?
[No] There isno such a potential negative societal impact in our work.
(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Assumptions 1, 2, 3 and 4.
(b) Did you include complete proofs of all theoretical results?
[Yes] See the subsectionA.1 in Appendix A.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] .
(b) Did you specify all the training details (e
g., data splits, hyperparameters, how theywere chosen)?
[Yes] .
(c) Did you report error bars (e
g., with respect to the random seed after running experiments multiple times)?
[Yes] .(d)
Did you include the total amount of compute and the type of resources used (e
g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] .
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[N/A]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A](c)
Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?
[N/A]
A Appendix
A.1 Theoretical Proofs
In the appendix, we provide all the technical details and proofs of the theorems stated in the paper.
Veriﬁcation of (1): Equation (1 holds because
[Equation]
where E p ∗ [ ψ (cid:48)
( R ( Z ))
R ∗ ( Z )]
=
E q ∗ [ ψ (cid:48)
( R ( Z ))] by the deﬁnition of R ∗ .
This veriﬁes (1).
We now prove the following lemmas.
Lemma A.1.
If the convex function f : R → R is µ -smooth over R , then for anyx, y ∈ R , the following inequality holds
[Equation]
Let f : R → R be a ﬁrst-order differentiable and convex function.
If f is σ -strongly convex,then for any x, y ∈ R , the following inequality holds
[Equation]
Proof of Lemma A.1.
The proof of Lemma A.1 is standard and can be found in Beck (2017).
Lemma A.2.
Under Assumptions 1 - 2 , we have(a).
There exist two constants c 0 = σe − 3 M, C 0 =
µe 3 M 2 , such that
[Equation]
and
[Equation]
(b).
For t 1 , t 2 ∈ [ − M, M ] , there exist two constants C 1 , C 2 , such that
[Equation]
and
[Equation]
Actually, we can take C 1 = 2 e 2 M µ, C 2 =
e M µ .
Proof of Lemma A.2.
(a) Let ∆ ψ ( x, y ) :
= ψ
[Equation]
x ) − ψ
[Equation]
y )
− ψ
[Equation]
x
[Equation]
( x − y
[Equation]
Since
E p ∗ ∆ ψ ( e D ( Z ) , e D ∗ ( Z ) )
=
B ψ ( e D ) −B ψ ( e D ∗ ) and ψ is µ -smooth and σ -strongly convex, by Lemma
A.1,σ
[Equation]
and then by Assumption 2,σe
[Equation]
As E p ∗ { D ( Z ) − D ∗ ( Z ) } 2 = E q ∗
e − D ∗ ( Z ) { D ( Z ) − D ∗ ( Z ) } 2 and (cid:107) D ∗ (cid:107) ∞ ≤ M , we have
[Equation]
Let c
=
σe − 3 M, C 0 =
µe 3 M 2 , then (A.1) and (A.2) imply that
[Equation]
(b) Obviously, for t 1 , t 2 ∈ [ − M, M ] ,
[Equation]
and
[Equation]
The proof of the lemma is completed
[Equation]
Proof of Theorem 1.
For notational convenience, denote (cid:15) n = (cid:107) D NN − D ∗ (cid:107) max and use E
I todenote E I ∗ , I = p, q .
Recall that E n I denotes the expectation with respect to (w.r.t) the empiricaldistribution of { Z I,t } n I t =1 for I = p, q .
As (cid:98) D ∈ arg min D ∈F FNN L n p ,n q ( D ) , where L n p ,n q ( D )
=1 /n p (cid:80) n p j =1 L 1 ( D ( Z p,j ))
+ 1 /n q (cid:80) n q i =
1 L 2 ( D ( Z q,i )) , we have
[Equation]
) − L
[Equation]
( D
[Equation]
(A.3)
By Theorem 2.1 in Bartlett et al.
(2005), with probability at least 1 − exp( − γ 1 ) ,
[Equation]
Also, with probability at least 1 − exp( − γ 1 ) ,
[Equation]
The inequalities (A.4) and (A.5) together imply that with probability at least 1 − 2 exp( − γ 1 ) ,
[Equation]
Step 1 .
Let g =
( D − D ∗ ) 2 , then g ≤ 4 M 2 by Assumption 2.
If (cid:107) D − D ∗ (cid:107) max ≤ r , then
[Equation]
Regarding g as a function of D − D ∗ , we have
[Equation]
Thus g can be viewed as the function of D − D ∗ with a Lipschitz constant 4 M .
Denote F D ∗ ,r FNN =
{ D ∈ F FNN , (cid:107) D − D ∗ (cid:107) max ≤ r } , and
[Equation]
where η Ii , i = 1 , 2 , . . .
, n I are i.i.d.
Rademacher variables.
For the rest of the proof of Theorem1, we use E η R n I F to denote the conditional expectation of R n I F w.r.t η Ii , i = 1 , 2 , . . .
, n I , given
Z I,i , i = 1 , 2 , . . . , n I and E I,η R n I F to denote the expectation of R n I F jointly w.r.t η Ii , Z I,i , i =1 , 2 , . . . , n I .
Again, by Theorem 2.1 in Bartlett et al.
(2005), with probability at least 1 − exp( − γ 1 ) ,
[Equation]
where the last inequality follows from Talagland’s contraction theorem.
Similarly, with probability atleast 1 − exp( − γ 1 ) ,
[Equation]
Let
R n ( r )
[Equation]
When
[Equation]
(A.7) and (A.8) indicate that with probability at least 1 − 2 exp( − γ 1 ) ,
[Equation]
≤ (2 r )
[Equation]
Thus, when (A.9) holds, with probability at least 1 − 2 exp( − γ 1 ) ,
[Equation]
D
[Equation]
p ,
[Equation]
Step 2 .
Suppose (cid:107) (cid:98) D − D ∗ (cid:107) max ≤ r 0 and let
G i =
[Equation]
L i ( D ) − L
i
( D ∗
[Equation]
: D ∈ F D ∗
[Equation]
r 0 FNN (cid:111) , i = 1 , 2 .
For each ( I, i ) ∈ { ( p, 1) , ( q, 2) } , with probability at least 1 − 2 exp( − γ 1 ) ,( E I − E n I ) {L i ( (cid:98) D ) − L
i
( D ∗ ) } ≤
[Equation]
E
[Equation]
R n I G i +
[Equation]
C i r(cid:114)
γ
[Equation]
n + 46 C
i Mγ
[Equation]
3 n .
(A.11)
Denote ˆ F D ∗ ,r FNN =
{ D ∈ F FNN , (cid:107) D − D ∗ (cid:107) n p ,n q ≤ r } .
By (A.10) in Step 1 , when r 20 ≥ R n ( r 0 ) andr 20 ≥ 16 M 2 γ 1 / (3 n ) , with probability at least 1 − 2 exp( − γ 1 ) , for each ( I, i ) ∈ { ( p, 1) , ( q, 2) } ,
E η
[Equation]
(cid:111) .
Denote ˆ F D ∗ ,r I =
{ D ∈ F FNN , (cid:107) D − D ∗ (cid:107) I,n I ≤ r } .
When n ≥ Pdim ( F FNN ) , r 0 ≥ 1 /n andn ≥ (2 eM ) 2 , we have
E η R n I { ( D − D ∗ ) : D ∈ ˆ F D ∗ , 2 r 0 I } ≤ 64 r(cid:114)
Pdim ( F FNN ) log n n , (A.12) and thus
[Equation]
Combining (A.3) (A.6) (A.11) and (A.13), with probability at least 1 − 8 exp( − γ 1 ) , we have
[Equation]
Therefore, when max (cid:110) (cid:112) Pdim ( F FNN ) log n
[Equation]
n,
[Equation]
n (cid:111) (cid:28) r 0 , there exists r 1 (cid:28) r
such that (cid:107) (cid:98) D −
D ∗ (cid:107) max (cid:28) r 1 .
Step 3 .
Let r ∗ =
inf { r ≥ :
R n ( s ) ≤ s 2 , for s ≥ r } and E = 533 (cid:110) (cid:107) D − D ∗ (cid:107) n p ,n q ≤ 4 r ∗ for all D ∈ F D ∗ , 2 r ∗ FNN (cid:111) .
We intend to prove
[Equation]
κM
[Equation]
(A.14)
When r ∗ ≤ 2 √ 3 M (cid:112) log n/n/ 3 , the inequality is trivial.
When r ∗ ≥ 2 √ 3 M (cid:112) log n/n/ 3 , by theresult in Step 1, P ( E ) ≥ 1 − 2 /n .
As a result,
[Equation]
For each I ∈ { p, q } ,
[Equation]
It follows from (A.12) that
[Equation] [Equation]
where κ = 24 × 130 .
Thus, r ∗ ≤ κM (cid:112) Pdim ( F FNN ) log n/n and (A.14) is proved.
Step 4 .
Let B max
( D ∗ , r ) = { D ∈ F FNN , (cid:107) D − D ∗ (cid:107) max ≤ r } , ¯ r ≥ max (cid:16)(cid:112) log n/n, r ∗ (cid:17) andl
= (cid:106) log 2 (2 M/ (cid:112) log n/n ) (cid:107) .
Then, the neural network function space F FNN can be divided into
B
[Equation] [Equation]
As ¯ r ≥ r ∗ , it then follows from the deﬁnition of r ∗ that ¯ r 2 ≥ R n (¯ r ) .
Further, if ¯ r 2 ≥16 M 2 γ 1 / (3 n ) , according to (A.10) in Step 1 , with probability at least 1 − 2 l exp( − γ 1 ) , for anyj = 1 , 2 , . .
, l ,
[Equation]
Suppose that for some j ≤ l , (cid:98) D ∈ B max
[Equation]
D ∗ , 2 j ¯ r )
[Equation]
B
[Equation]
D ∗ , 2 j − 1 ¯ r ) , then by the results in Step 2,with probability at least 1 − 8 exp( − γ 1 ) ,
[Equation]
If
[Equation]
15) and
[Equation]
then
[Equation]
In short, to obtain this inequality, we need ¯ r satisfying (A.15), (A.16) and ¯ r ≥max (cid:16)(cid:112) log n/n, 4 √ 3 M (cid:112)
γ 1 /n/ 3 , r ∗ (cid:17) .
As r ∗ ≤ κM (cid:112) Pdim ( F FNN ) log n/n , there exists a constant C ∗ =
C ∗ ( c 0 , C 0 , C 1 , C 2 , M ) = C (cid:48)
( µ, σ, M ) such that
[Equation]
satisﬁes all the requirements.
As a result, with probability at least 1 − 10 l exp( − γ 1 ) ,(cid:107) (cid:98) D − D ∗ (cid:107) max ≤ ¯ r and (cid:107) (cid:98) D
− D ∗ (cid:107) n p ,n q ≤ 2¯ r.
Let γ 1 = log 10 l + γ, l = (cid:98) log 2 (2 M/ (cid:112) log n/n ) (cid:99) , there exists C = C
( c 0 , C 0 , C 1 , C 2 , M ) =
C ( µ, σ, M ) such that with probability at least 1 − exp( − γ ) ,
[Equation]
∗
[Equation]
max
[Equation]
(cid:32)(cid:114)
[Equation]
n
[Equation]
n
[Equation]
, and
[Equation]
(cid:32)(cid:114)
[Equation]
n
[Equation]
The proof of Theorem 1 is completed.
Lemma A.3.
The following excess risk decomposition always holds:
[Equation]
Under Assumptions 1 and 3 , when n ≥ Pdim ( F FNN ) , there exist three constants C, C 0 , C ∗ , with
C, C 0 depending only on ( µ, σ, M ) and C ∗ depending only on ( µ, σ ) , such that
[Equation]
and
[Equation]
Proof of Lemma A.3.
To show (A.18) is the key step in the proof of this theorem, thus we focus onthe proof of (A.18).
Let
[Equation]
Then,
[Equation]
By the symmetrization technique, Talagrand’s lemma, (A.12) and the fact that (cid:107) D (cid:107) ∞ ≤ M for any
D ∈ F FNN , we can easily get the inequality (A.18) through (A.19).
Proof of Theorem 2.
Theorem 2 is a direct corollary of Lemma A.3.
We omit the details here.
Proof of Theorem 3.
Since D ∗ ∈ H β ([0 , 1] d , M ) with β
= k + a where k ∈ N + and a ∈ (0 , 1] ,by Lemma 1, for the F FNN , a function class consists of ReLU FNN with width W = 38( (cid:98) β (cid:99) +1) 2 d (cid:98) β (cid:99) +1 L (cid:100) log 2 (8 L ) (cid:101) and depth D =
( (cid:98) β (cid:99) + 1) 2 K (cid:100) log 2 (8 K ) (cid:101) , where K, L ∈ N + , there existsa function φ 0 ∈ F FNN such that
[Equation]
where H B,δ = ∪ d i =1 (cid:8) x
= [ x 1 , . . .
, x d ] : x i ∈ ∪ B
[Equation]
1 b =1 ( b/B − δ, b/B ) (cid:9) , B = (cid:100) ( KL ) 2 /d (cid:101) , δ ∈(0 , 1 / (3 B )] .
As D NN ∈ arg min D ∈F FNN (cid:107) D − D ∗ (cid:107) max , then
[Equation]
By the result in (A.20), for I = p or q , we have
[Equation]
As p ∗ ( · ) , q ∗ ( · ) are the density functions of some measures on [0 , 1] d which are absolutely continuouswith respect to the Lebesgue measure and δ can be arbitrarily small, (cid:82) H B,δ I 0 ( x ) dx is also arbitrarilysmall.
Thus we have
[Equation]
and
[Equation]
By Corollary 1, there exists a constant C 1 only depending on ( µ, σ, M ) such that
[Equation]
This completes the proof of the ﬁrst part of Theorem 3.
As for the second part of this theorem, based on Theorem 6 in Bartlett et al.
(2019), for a speciﬁc
ReLU network f φ , where φ contains the parameters in the network, there exists a universal constant
C 2 such that
[Equation]
where S is the total number of parameters in the network f φ .
For a ReLU FNN with width
W and depth D , it can be easily checked that S
= O ( W 2 D ) .
Now for W = 114( (cid:98) β (cid:99) +1) 2 d (cid:98) β
(cid:99) +1 , D =
( (cid:98) β (cid:99) + 1) 2 (cid:108) n
[Equation] [Equation]
d +
[Equation]
) log 2 (cid:16)
[Equation]
n d 2( d +2 β ) (cid:17)(cid:109) , and W , D satisfy O ( W 2 D ) =
O (cid:16) ( (cid:98) β (cid:99) + 1) 6 d 2 (cid:98)
β (cid:99) +2 (cid:108) n
[Equation]
[Equation]
d +2 β ) log − 3 n (cid:109)(cid:17) , which means L = 1 , K = (cid:108) n d 2( d +2 β ) (cid:109) , and there existthree universal constants C 3 , C 4 , C 5 such that
[Equation]
(cid:99) + 1) log d
[Equation]
d
[Equation]
d
[Equation]
log n (cid:27) ≤
C
[Equation]
n
[Equation]
n (cid:111) { 6( (cid:98) β
(cid:99)
+ 1) + 2( (cid:98) β (cid:99)
+ 1) d
+ log n }
[Equation]
C
[Equation]
( (cid:98)
[Equation]
(cid:99) + 1) 9 d 2 (cid:98) β
(cid:99)
+
[Equation]
n − 2 β d +2 β .
(A.22)
It follows from (A.21)
that
[Equation]
This completes the proof of the second part of Theorem 3.
Proof of Theorem 4.
Based on Theorem 3.1 in Baraniuk & Wakin (2009), there exists a linearprojection A ∈ R d δ × d such that AA T = dI
d δ /d δ , where I d δ ∈ R d δ × d δ is an identity matrix, and forany x, y ∈ M ,
[Equation]
Then we have
[Equation]
Note that for any z ∈ A ( M ) , there exits a unique x ∈ M such that z
= Ax .
Otherwise, suppose wecan ﬁnd x, x (cid:48) ∈ M , x (cid:54) =
x (cid:48)
such that z
=
Ax = Ax (cid:48) , then by (A.23), we know (cid:107) x − x (cid:48) (cid:107) 2 = 0 andthus x = x (cid:48) , which contradicts the assumption that x (cid:54) =
x (cid:48) .
This uniqueness allows us to deﬁne alinear operator SL : A ( M ) → M such that A [ SL ( z )]
=
z .
By (A.23), we have
[Equation]
This implies that the norm of SL belongs to [1 / (1 + δ ) , 1 / (1 − δ )] .
For the high-dimensionalfunction D ∗ : [0 , 1] d → R whose support is M ρ , it has a approximate low-dimensional representation˜ D ∗ as follows:
[Equation]
As D ∗ ∈ H β
([0 , 1] d , M ) with β = k
+ a where k ∈ N + and a ∈ (0 , 1] , we have˜
D ∗ ∈ H β (cid:0) A ( M ) , M/ (1 − δ ) β (cid:1) .
By the extended version of Whitney’s extension theoremin Fefferman (2006), since A ( M ) ⊆
A (cid:0) [0 , 1] d (cid:1) ⊆ (cid:2) − d/ √ d δ , d/ √ d δ (cid:3) d δ , there exists ˜ D ∗ E ∈
H β (cid:16)(cid:2) − d/ √ d δ , d/ √ d δ (cid:3) d δ , M/ (1 − δ ) β (cid:17)
such that ˜ D ∗ E ≡ ˜ D ∗ on A ( M ) .
If W = 38( (cid:98) β (cid:99) +1)
d δ (cid:98) β
(cid:99)
+1 L (cid:100) log 2 (8 L ) (cid:101) and D =
( (cid:98) β (cid:99) + 1) 2 K (cid:100) log 2 (8 K ) (cid:101) , by the ﬁrst result of Lemma 1,there exists a function φ 0 implemented by a ReLU network with width W and depth D such thatsup
[Equation]
1) 2 (2 d ) β d (cid:98) β (cid:99) +( β ∨ 1+ β ) / 2 δ ( KL ) − 2 βdδ .
where H d δ B,(cid:15)
=
∪ d δ i =1 (cid:8) x
= [ x 1 , x 2 , . .
, x d δ ] : x i ∈ ∪ B − 1 b =1 ( b/B − (cid:15), b/B ) (cid:9) and B =(cid:100) ( KL ) 2 /d
(cid:101) , (cid:15) ∈ (0 , 1 / (3 B )] .
Thussup
[Equation]
(cid:19)(cid:12)(cid:12)(cid:12)(cid:12) ≤ 18 M (1 − δ ) β ( (cid:98) β (cid:99) + 1)
[Equation]
2 d )
[Equation]
d (cid:98) β
[Equation]
+
[Equation]
δ ( KL ) − 2 βdδ , where ˜ H d δ B,(cid:15)
= (cid:110) (2 dt − d 1 d δ ) / √ d
δ : t ∈ H d δ B,(cid:15) (cid:111) .
Let ˜ φ 0 ( x ) = φ 0 (cid:0)
[Equation]
√
[Equation]
δ Ax +
[Equation]
d δ ) /
[Equation]
d )
[Equation]
and ¯ H d ∗ B,(cid:15)
= 611 (cid:110) x ∈ [0 , 1] d × d : ( √ d δ Ax + d 1 d δ ) / (
2 d ) ∈
H
d δ B,(cid:15) (cid:111) .
It can be easily checked that ˜ φ 0 is also a functionimplemented by a ReLU network with the same structure as φ 0 , except that the input layer of ˜ φ 0 hasd units, instead of d δ units.
For any x ∈ M ρ \ ¯ H d ∗ B,(cid:15) , Ax ∈ (cid:2) − d/ √ d δ , d/ √ d δ (cid:3) d δ \ ˜ H
d δ B,(cid:15) and thereexists a x
(cid:48) ∈
M satisfying (cid:107) x − x (cid:48)
(cid:107) 2 ≤ ρ .
Since ˜ D ∗ E ∈ H β (cid:16)(cid:2) − d/ √ d δ , d/ √ d δ (cid:3) d δ , M/ (1 − δ ) β (cid:17) and D ∗ ∈ H β (
[Equation]
0 , 1] d , M ) ,
[Equation]
where the last inequality holds when ρ
[Equation]
By the result in (A.24), for I = p or q , it holds
[Equation]
As p ∗ ( · ) , q ∗ ( · ) are the density functions of some measures on [0 , 1] d which are absolutely continuousw.r.t the Lebesgue measure and (cid:15) can be arbitrarily small for the given δ , (cid:82)
¯ H d ∗ B,(cid:15)
I 0 ( x ) dx is alsoarbitrarily small for the given δ .
Thus we have
[Equation]
and
[Equation]
By Corollary
[Equation]
there exists a constant C 1 only depending on ( µ, σ, M )
[Equation]
such that
[Equation]
For
[Equation] [Equation]
and W , D satisfy
[Equation]
along the derivation of (A.22), there exists a universal constants C ∗ such that
[Equation]
Based on the result of (A.25),
[Equation]
This completes the proof of the theorem and (12).
Proof of Proposition 1.
For k = 0 , . . .
, K − 2 , the densities q k () , q k +1 () of the synthetic data{ Z k,j } nj =1 and { Z k +1 ,j } nj =1 satisfyq k ( t )
[Equation]
As (cid:107) f (cid:107)
=
( (cid:82) Z f 2 ( x ) dx ) 12 , then for any density g satisfying g ≥ c , (cid:107) f (cid:107) 2 = (cid:0)(cid:82)
Z f 2 ( x ) dx (cid:1) 12 ≤(cid:0)(cid:82)
Z f 2 ( x ) g
( x ) /cdx
(cid:1) 12 = (cid:107) f (cid:107) g / √ c .
Using an appropriate F 0 FNN whose element D satisﬁes (cid:107) D (cid:107) ∞ ≤
M 0 , for the direct estimate (cid:98) D SRE , as log( q ∗ /p ∗ ) is only bounded from below by − M 0 , by Theorem2, we have
[Equation]
For k = 0 , 1 , . . .
, K − 2 , as | log { q k ( t ) /q
k +1 ( t ) }| is bounded by M 0 , by Corollary 1, we have
[Equation]
Let R ∗ K − 1 ,M
= (1 − α K − 1 ) R ∗ M
+ α K − 1 .
As the logarithm of R ∗ K − 1 = (1 − α K − 1 ) q ∗ /p
∗
+ α K − 1is also only bounded from below by − M 0 , again, by Theorem 2,
[Equation]
Thus
[Equation]
This completes the proof of Proposition 1.
A.2 Examples of Hölder function class
Let p ∗ be the density function of a truncated d -dimensional multivariate Gaussian with mean zeroand covariance matrix
Σ p ∈ R d × d in [0 , 1] d .
That means
[Equation]
Similarly, let
[Equation]
for some positive deﬁnite matrix Σ q .
For any matrix A ∈ R d × d , A i, · is the i th row of A fori = 1 , 2 , . . .
, d and
[Equation]
Then,
D ∗ ( z ) =
[Equation]
z )
[Equation]
Let M = max
[Equation]
It is straightforward to check that
[Equation]
It implies the Hölder smoothness parameter β is 2 for this example.
Moreover, the truncated multivariate Gaussian distributions considered above are special cases of theexponential distribution class deﬁned below.
Deﬁne the density function class
Exp ( β, B ) :=
[Equation]
Suppose that Σ ∈ R d × d is positive deﬁnite and let g ( z )
=
z (cid:48)
Σ z/ 2 .
Then, g ∈ H 2 ([0 , 1] d , M Σ ) ,where M Σ = max (cid:8) 12 ( (cid:107) Σ 1 / 2 (cid:107) 22 , ∞ , (cid:107) Σ i, · (cid:107) 2 , i = 1 , 2 , . . .
, d (cid:9) .
If p ∗ , q ∗ ∈ Exp ( β, B ) , we have
D ∗ ( z ) = log[ q ∗ ( z )
/p
∗ ( z )]
∈ H β
[0 , 1] d , 4 B ) .
A.3 Extension to unbounded support case
In fact, our Theorem 1, Corollary 1 and Theorem 2 do not rely on the hypercube assumption.
Torelax the hypercube assumption to allow unbounded support, we need to study the upper boundfor the approximation error (cid:107) D NN − D ∗ (cid:107) max carefully.
With unbounded support, we may bound(cid:107) D NN − D ∗ (cid:107) max by the truncation technique under some additional assumptions, at a small price ofan additional logarithm term in the error bound.
Speciﬁcally, when the pdfs are supported on R d , to bound the approximation error as in Theorem3, aside from Assumptions 1-2 and the Hölder class assumption, we need to further assume thatmax { E p ∗ I ( (cid:107)
Z
[Equation]
∞ ≥ log n
[Equation]
, E
[Equation]
Z
[Equation]
log n ) } ≤ n − 2 β d +2 β .
For I = p or q , and any D ∈ F FNN ,where F FNN is the function class of ReLU FNNs with width W and depth D speciﬁed by
[Equation]
we have
[Equation]
where the second inequality follows from the facts that (cid:107) D ∗ (cid:107) ∞ ≤ M, (cid:107) D (cid:107) ∞
≤ M under Assumption2.
Since D ∗ ∈ H β ( R d , M ) , D ∗ (2 t log n − log n 1 d ) ∈
H β ([0 , 1] d , (2 log n ) (cid:98) β
(cid:99) M )
as a function oft , where 1 d is the d -dimensional all-one vector.
By Lemma 1, there exists a function φ 0 ∈ F FNN suchthat
[Equation]
[Table]
where H B,δ = ∪ di =1 (cid:8) t =
[ t 1 , . . . ,
t d ] : t i ∈ ∪ B − 1 b =1 ( b/B − δ, b/B ) (cid:9) , B = (cid:100) n 1 d +2 β (cid:101) , δ ∈(0 , 1 / (3 B )] .
Thussup
[Equation]
where ˜ H dB,δ = (cid:110) 2 t log n − log n : t ∈ H dB,δ (cid:111) .
Let ˜ φ 0 ( z )
=
φ 0 (cid:16) z +log n 1 d 2log n (cid:17) ∈ F FNN .
As δ can bearbitrarily small, it then follows from similar lines as in the proof of Theorem 3 that
E
[Equation]
Since D NN ∈ arg min D ∈F FNN (cid:107) D − D ∗ (cid:107) max , we have(cid:107)
D NN − D ∗ (cid:107) 2 max ≤ (cid:107)
[Equation]
Compared with the upper bound of the approximation error in Theorem 3, when the pdfs are supportedon R d (unbounded case), a similar approximation error upper bound can be derived with an additionallogrithmic factor (2 log n ) 2 (cid:98) β (cid:99) .
A.4 Simulation setting and implementation details
Our simulation settings are as follows.
• Beta setting: Let Z
= ( Z 1 , Z 2 , . .
, Z p )
(cid:62) ∈ R p be a random vector of interest, whereZ 1 , Z 2 , . . .
, Z p are i.i.d.
random variables following
Beta
distribution, denoted by
Beta ( α, β ) .
Set p ∗ as the p.d.f of Beta (2 . 2 , 1 . 5) and q ∗ as the p.d.f of Beta (2 , 2) .
In
this setting, we set p = 5 .•
Normal setting: Let Z =
( Z 1 , Z 2 , . .
, Z d , Z d +1 , Z d +2 , . . .
, Z 2 d ) (cid:62) ∈ R 2 d be some random
vector of interest.
Let p ∗ be the p.d.f of N (0 , I 2 d ) and q ∗ be the p.d.f of N (0 , Σ( ρ )) , where
Σ( ρ ) =
( σ ρi,j
) ∈ R 2 d × 2 d and
σ ρi,j
[Equation]
In this setting, we set d = 5 and ρ = 0 .
We apply the Adam algorithm (Kingma & Ba, 2014) in Pytorch with a learning rate lr = 0 . 0001 anda weight decay parameter wd = 0 . 0001 .
A neural network with 2 hidden layers with widths (64 , 64)and ReLU activation function, is used in the experiment.
The maximum number of epoches is 20000 .
In this experiment, the training data size n is 5000 ( 10000 ).
A validation data is used.
The batch sizeis 500 ( 1000 ), and an early-stopping technique is applied with patience = 100 for Beta setting andpatience = 1000 for Normal setting, where patience is the number of epochs until termination if noimprovement is made on the validation dataset.
The experiment is conducted with a laptop with an
Intel(R) Core(TM) i7-8750H @
2.20GHz CPU having 6 cores.
We use the LR-Bregman divergence inthis example.
For the sequence
= α 0 < α 1 < · · · < α K − 1 < α K = 1 , we use the linearly spacedα k ’s, that is α k = k/K , k = 0 , 1 , 2 , . .
, K .