L EARNING WITH S TOCHASTIC O RDERS
yzs2@ edu Youssef Mroueh IBM Research AI mroueh@us.ibm.com
A BSTRACT
Learning high-dimensional distributions is often done with explicit likelihood modeling or implicit modeling via minimizing integral probability metrics (IPMs).
In this paper, we expand this learning paradigm to stochastic orders, namely, the convex or Choquet order between probability measures
Towards this end , exploiting the relation between convex orders and optimal transport, we introduce the Choquet-Toland distance between probability measures, that can be used as a drop-in replacement for IPMs.
We also introduce the Variational Dominance
Criterion (VDC) to learn probability measures with dominance constraints, that encode the desired stochastic order between the learned measure and a known baseline.
We analyze both quantities and show that they suffer from the curse of dimensionality and propose surrogates via input convex maxout networks (ICMNs), that enjoy parametric rates.
We provide a min-max framework for learning with stochastic orders and validate it experimentally on synthetic and high-dimensional image generation, with promising results.
Finally, our ICMNs class of convex functions and its derived Rademacher Complexity are of independent interest beyond their application in convex orders.
The code is available at https:// github.com/yair-schiff/stochastic-orders
ICMN
I NTRODUCTION
Learning complex high-dimensional distributions with implicit generative models (Goodfellow et al., 2014; Mohamed & Lakshminarayanan, 2017; Arjovsky et al., 2017) via minimizing integral probability metrics (IPMs) (M ¨ uller, 1997a) has led to the state of the art generation across many data modalities (Karras et al., 2019; De Cao & Kipf, 2018;
Padhi et al., 2020).
An IPM compares probability distributions with a witness function belonging to a function class F , e.g., the class of
Lipchitz functions, which makes the IPM correspond to the Wasserstein distance 1.
While estimating the witness function in such large function classes suffers from the curse of dimensionality, restricting it to a class of neural networks leads to the so called neural net distance (Arora et al., 2017) that enjoys parametric statistical rates.
In probability theory, the question of comparing distributions is not limited to assessing only equality between two distributions.
Stochastic orders were introduced to capture the notion of dominance between measures.
Similar to IPMs, stochastic orders can be defined by looking at the integrals of measures over function classes F (M¨uller, 1997b).
Namely, for µ + , µ − ∈ P 1 ( R d ) , µ + dominates µ − , or µ − ⪯ µ + , if for any function f ∈ F , we have (cid:82) R d f ( x ) dµ − ( x ) ≤ (cid:82) R d f ( x ) dµ + ( x ) (See
Figure 1a for an example).
In the present work, we focus on the Choquet or convex order (Ekeland &
Schachermayer, 2014) generated by the space of convex functions (see Sec.
2 for more details).
Previous work has focused on learning with stochastic orders in the one dimensional setting, as it has prominent applications in mathematical finance and distributional reinforcement learning (RL).
The survival function gives a characterization of the convex order in one dimension (See Figure 1b and Sec. 2 for more details).
For instance, in portfolio optimization (Xue et al., 2020; Post et al., 2018; Dentcheva & Ruszczynski, 2003)
the goal is to find the portfolio that maximizes the expected return under dominance constraints between the return distribution and a benchmark distribution.
[Figure]
A similar concept was introduced in distributional RL (Martin et al., 2020) for learning policies with dominance constraints on the distribution of the reward.
While these works are limited to the univariate setting, our work is the first, to the best of our knowledge, that provides a computationally tractable characterization of stochastic orders that is sample efficient and scalable to high dimensions.
The paper is organized as follows: in Sec.
3 we introduce the Variational Dominance Criterion (VDC); the VDC between measures µ + and µ − takes value
if and only if µ + dominates µ − in the convex order, but it suffers from the curse of dimension and cannot be estimated efficiently from samples.
To remediate this, in Sec. 4 we introduce a VDC surrogate via Input Convex Maxout
Networks (ICMNs).
ICMNs are new variants of Input Convex Neural Nets (Amos et al., 2017) that we propose as proxy for convex functions and study their complexity.
We show in Sec. 4 that the surrogate VDC has parametric rates and can be efficiently estimated from samples.
The surrogate
VDC can be computed using (stochastic) gradient descent on the parameters of the ICMN and can characterize convex dominance (See Figure 1c).
We then show in Sec.
5 how to use the VDC and its surrogate to define a pseudo-distance on the probability space.
Finally, in Sec. 6 we propose penalizing generative models training losses with the surrogate VDC to learn implicit generative models that have better coverage and spread than known baselines.
This leads to a min-max game similar to GANs.
We validate our framework in Sec.
7 with experiments on portfolio optimization and image generation.
T HE C HOQUET OR CONVEX ORDER
Denote by P ( R d ) the set of Borel probability measures on R d and by P 1 ( R d ) ⊂ P ( R d ) the subset of those which have finite first moment: µ ∈ P 1 ( R )
if and only if (cid:82) R d ∥ x ∥ dµ ( x ) < + ∞ .
Comparing probability distributions Integral probability metrics (IPMs) are pseudo-distances between probability measures µ, ν defined as d F ( µ, ν )
= sup f ∈F E µ f − E ν f , for a given function class F which is symmetric with respect to sign flips.
They are ubiquitous in optimal transport and generative modeling to compare distributions: if F is the set of functions with Lipschitz constant 1, then the resulting IPM is the 1-Wasserstein distance; if F is the unit ball of an RKHS, the IPM is its maximum mean discrepancy.
Clearly, d F ( µ, ν ) =
if and only E µ f = E ν f for all f ∈ F , and when F is large enough, this is equivalent to µ = ν .
The Choquet or convex order When the class F is not symmetric with respect to sign flips, comparing the expectations E µ f and E ν f for f ∈ F does not yield a pseudo-distance.
In the case where F is the set of convex functions, the convex order naturally arises instead:
Definition 1 (Choquet order, Ekeland & Schachermayer (2014), Def. 4) .
For µ − , µ + ∈ P 1 ( R d ) , we say that µ − ⪯ µ + if for any convex function f : R d → R , we have
[Equation]
µ − ⪯ µ + is classically denoted as “ µ − is a balay´ee of µ + ”, or “ µ + dominates µ − ”.
It turns out that ⪯ is a partial order on P 1 ( R d ) , meaning that reflexivity ( µ ⪯ µ ), antisymmetry (if µ ⪯ ν and ν ⪯ µ , then µ = ν ), and transitivity (if µ 1 ⪯ µ 2 and µ 2 ⪯ µ 3 , then µ 1 ⪯ µ 3 ) hold.
As an example, if µ − , µ + are Gaussians µ − = N (0 , Σ − ) , µ + = N (0 , Σ + ) , then µ − ⪯ µ + if and only if Σ − ⪯ Σ + in the positive-semidefinite order (M¨uller, 2001).
Also, since linear functions are convex, µ − ⪯ µ + implies that both measures have the same expectation: E x ∼ µ − x = E x ∼ µ + x .
In the univariate case, µ − ⪯ µ + implies that supp( µ − ) ⊆
supp( µ + ) and that Var( µ − )
≤ Var( µ + ) , and we have that µ − ⪯ µ + holds if and only if for all x ∈ R , (cid:82) + ∞ x
[Equation]
where ¯ F · is the survival function (one minus the cumulative distribution function).
Note that this characterization can be checked efficiently if one has access to samples of µ − and µ + .
In the high-dimensional case, there exists an alternative characterization of the convex order:
Proposition 1 (Ekeland & Schachermayer (2014), Thm. 10) .
If µ − , µ + ∈ P 1 ( R d ) , we have µ − ⪯ µ + if and only if there exists a Markov kernel R (i.e. ∀ x ∈ R d , (cid:82) R d y dR x ( y ) = x ) such that µ + = (cid:82) R d R x dµ − .
Equivalently, there exists a coupling ( X, Y ) such that Law( X ) = µ − , Law( Y ) = µ + and X =
E ( Y | X ) almost surely.
Intuitively, this means that µ + is more spread out than µ − .
Remark that this characterization is difficult to check, especially in high dimensions.
T HE V ARIATIONAL D OMINANCE C RITERION
In this section, we present a quantitative way to deal with convex orders.
Given a bounded open convex subset Ω ⊆ R d and a compact set K ⊆ R d , let A = { u : Ω → R , u convex and ∇ u ∈
K almost everywhere } .
We define the Variational Dominance Criterion (VDC) between probability measures µ + and µ − supported on Ω analogously to IPMs, replacing F by A :
[Equation]
Remark that when 0 ∈ K , VDC A ( µ + || µ − ) ≥ 0 because the zero function belongs to the set A .
We reemphasize that since A is not symmetric with respect to sign flips as f ∈ A does not imply − f ∈ A , the properties of the VDC are very different from those of IPMs.
Most importantly, the following proposition, shown in App.
A, links the VDC to the Choquet order.
Proposition 2.
Let K compact such that the origin belongs to the interior of K .
If µ + , µ − ∈ P (Ω) ,
VDC A ( µ + || µ − ) :
= sup u ∈A (cid:82) Ω u d ( µ − − µ + ) =
if and only if (cid:82) Ω u d ( µ − − µ + ) ≤ 0 for any convex function on Ω (i.e. µ − ⪯ µ + according to the Choquet order).
That is, Proposition 2 states that the VDC between µ + and µ − takes value
if and only if µ + dominates µ − .
Combining this with the interpretation of Proposition 1, we see that intuitively, the quantity VDC A ( µ + || µ − ) is small when µ + is more spread out than µ − , and large otherwise .
Hence, if we want to enforce or induce a Choquet ordering between two measures in an optimization problem, we can include the VDC (or rather, its surrogate introduced in Sec. 4) as a penalization term in the objective.
Before this, we explore the connections between VDC and optimal transport, and study some statistical properties of the VDC.
3. T HE VDC AND OPTIMAL TRANSPORT
Toland duality provides a way to interpret the VDC through the lens of optimal transport.
In the following, W 2 ( µ, ν ) denotes the 2-Wasserstein distance between µ and ν .
Theorem 1 (Toland duality, adapted from Thm.
1 of Carlier (2008)) .
For any µ + , µ − ∈ P (Ω) , the
VDC satisfies:
[Equation]
The optimal convex function u of VDC in (1) and the optimal ν in the right-hand side of (2) satisfy ( ∇ u ) # µ
+ = ( ∇ u ) # µ − =
ν , where ( ∇ u ) # µ + denotes the pushforward of µ + by ∇ u .
Note that under the assumption 0 ∈ K , Theorem 1 implies that VDC A ( µ + || µ − )
=
if and only if W 22 ( µ + , ν ) − 12 (cid:82) Ω ∥ x ∥ 2 dµ + ≤ W 22 ( µ − , ν ) − 12 (cid:82) Ω ∥ x ∥ 2 dµ − for any ν ∈ P ( K ) .
Under the equivalence VDC A ( µ + || µ − ) =
⇐⇒ µ + ⪰ µ − shown by Proposition 2, this provides yet another characterization of the convex order for arbitrary dimension.
3. S TATISTICAL RATES FOR VDC ESTIMATION
In this subsection, we present an upper bound on the statistical rate of estimation of VDC A ( µ || ν ) using the estimator VDC A ( µ n || ν n ) based on the empirical distributions µ n = 1 n (cid:80) ni
=
δ x i , ν n =n (cid:80) ni
=1 δ y i built from i.i.d.
samples ( x i ) ni =1 , ( y i ) ni
=1 from µ and ν , respectively.
Theorem 2.
Let Ω =
[ − 1 , 1] d
and K = { x ∈ R d | ∥ x ∥ 2 ≤ C } for an arbitrary C > 0 .
With probability at least 1 − δ ,
[Equation]
where K depends on C and d .
The proof of this result is in App. A.
The dependency on n − 2 d is indicative of the curse of dimension: we need a number of samples n exponential in d to control the estimation error.
While Theoremonly shows an upper bound on the difference between the VDC and its estimator, in Subsec.
5.1 we study a related setting where a ˜Ω( n − 2 d ) lower bound is available.
Hence, we hypothesize that VDC estimation is in fact cursed by dimension in general.
A VDC SURROGATE VIA INPUT CONVEX MAXOUT NETWORKS
Given the link between the VDC and the convex order, one is inclined to use the VDC as a quantitative proxy to induce convex order domination in optimization problems.
Estimating the VDC implies solving an optimization problem over convex functions.
In practice, we only have access to the empirical versions µ n , ν n of the probability measures µ, ν ; we could compute the VDC between the empirical measures by solving a linear program similar to the ones used in non-parametric convex regression (Hildreth, 1954).
However, the statistical rates for the VDC estimation from samples are cursed by dimension (Subsec. 3.2), which means that we would need a number of samples exponential in the dimension to get a good estimate.
Our approach is to focus on a surrogate problem instead:
[Equation]
where ˆ A is a class of neural network functions included in A over which we can optimize efficiently.
In constructing ˆ A , we want to hardcode the constraints u convex and ∇ u ∈ K almost everywhere into the neural network architectures.
A possible approach would be to use the input convex neural networks (ICNNs) introduced by Amos et al.
(2017), which have been used as a surrogate of convex functions for generative modeling with normalizing flows (Huang et al., 2021)
in optimal transport (Korotin et al., 2021a;b; Huang et al., 2021; Makkuva et al., 2020) and large-scale Wasserstein flows (Alvarez-Melis et al., 2021; Bunne et al., 2021; Mokrov et al., 2021).
However, we found in early experimentation that a superior alternative is to use input convex maxout networks (ICMNs), which are maxout networks (Goodfellow et al., 2013) that are convex with respect to inputs.
Maxout networks and ICMNs are defined as follows:
Definition 2 (Maxout networks) .
For a depth L ≥ 2 , let M =
( m 1 , . . .
, m L ) be a vector of positive integers such that m 1 = d .
Let F L, M ,k be the space of k -maxout networks of depth L and widths M , which contains functions of the form
[Equation]
where for any 2 ≤ ℓ ≤ L − 1 , and any 1 ≤ i ≤ m ℓ , the i -th component of x ( ℓ ) = ( x ( ℓ ) 1 , . .
, x ( ℓ ) m ℓ ) is computed recursively as:
[Equation]
with x (1) = x .
Definition 3 (Input convex maxout networks or ICMNs) .
A maxout network f of the form (3) - (4) is an input convex maxout network if (i) for any 1 ≤ i ≤ M L , a i ≥ 0 , and (ii) for any 2 < ℓ ≤ L − 1 , 1 ≤ i ≤ m ℓ +1 , 1 ≤ j ≤ k , the first m ℓ components of w ( ℓ ) i,j are non-negative.
We denote the space of
ICMNs as F L, M ,k, + .
In other words, a maxout network is an ICMN if all the non-bias weights beyond the first layer are constrained to be non-negative.
This definition is analogous to the one of ICNNs in Amos et al.
(2017), which are also defined as neural networks with positivity constraints on non-bias weights beyond the first layer.
Proposition 5 in App.
B shows that ICMNs are convex w.r.t to their inputs.
It remains to impose the condition ∇ u ∈ K almost everywhere , which in practice is enforced by adding the norms of the weights as a regularization term to the loss function.
For theoretical purposes, we define F L, M ,k (1) (resp.
F L, M ,k, + (1) )
as the subset of F L, M ,k (resp.
F L, M ,k, + ) such that for all 1 ≤ ℓ ≤ L − 1 , 1 ≤ i ≤ m ℓ , 1 ≤ j ≤ k , ∥ w ( ℓ ) i,j ∥ 2 ≤ 1 , and ∥ a ∥ 2 = (cid:80) m
L i =1 a 2 i ≤ 1 .
The following proposition, proven in App. B, shows simple bounds on the values of the functions in F L, M ,k (1) and their derivatives.
Proposition 3.
Let f be an ICMN that belongs to F L, M ,k (1) .
For x almost everywhere in B 1 ( R d ) , ∥∇ f ( x ) ∥ ≤ 1 .
Moreover, for all x ∈ B 1 ( R d ) , | f ( x ) | ≤ L , and for 1 ≤ ℓ ≤ L , ∥ x ( ℓ ) ∥ ≤ ℓ .
When K = B 1 ( R d ) , we have that the space of ICMNs F L, M ,k, + (1) is included in A .
Hence, we define the surrogate VDC associated to F L, M ,k, + (1) as:
[Equation]
Theorem 3.
Suppose that for all 1 ≤ ℓ ≤ L , the widths m ℓ satisfy m ℓ ≤ m , and assume that µ, ν are supported on the ball of R d of radius r .
With probability at least 1 − δ ,
[Equation]
We see from Theorem 3 that VDC F L, M ,k, + (1) in contrast to VDC A has parametric rates and hence favorable properties to be estimated from samples.
In the following section, wes see that VDC can be used to defined a pseudo-distance on the probability space.
F ROM THE CONVEX ORDER BACK TO A PSEUDO - DISTANCE
We define the Choquet-Toland distance (CT distance) as the map d CT , A : P (Ω) × P (Ω) → R given by d [Equation] |
[Equation]
That is, the CT distance between µ + and µ − is simply the sum of Variational Dominance Criteria.
Applying Theorem 1, we obtain that d CT , A ( µ + , µ −
[Equation]
=
[Equation]
sup ν
[Equation]
22 ( µ + , ν ) − W 22 ( µ − , ν ) (cid:9) + sup ν ∈P ( K ) (cid:8) W 22 ( µ + , ν ) − W 22 ( µ − , ν ) (cid:9) ) .
The following result, shown in App. C, states that d CT,K is indeed a distance.
Theorem 4.
Suppose that the origin belongs to the interior of K .
d CT , A is a distance, i.e. it fulfills
(i)
d CT , A ( µ + , µ − ) ≥ 0 for any µ + , µ − ∈ P (Ω) (non-negativity).
(ii) d CT , A ( µ + , µ − ) =
if and only if µ + = µ − (identity of indiscernibles).
(iii)
If µ 1 , µ 2 , µ 3 ∈ P (Ω) , we have that d CT , A ( µ 1 , µ 2 ) ≤ d CT , A ( µ 1 , µ 3 ) + d CT , A ( µ 3 , µ 2 ) .
As in (5), we define the surrogate CT distance as:
[Equation]
5. S TATISTICAL RATES FOR CT DISTANCE ESTIMATION
We show almost-tight upper and lower bounds on the expectation of the Choquet-Toland distance between a probability measure and its empirical version.
Namely, E [ d CT , A ( µ, µ n )] =
O ( n − 2 d ) , Ω( n − 2 d / log( n )) .
Theorem 5.
Let C 0 , C 1 be universal constants independent of the dimension d .
Let Ω =
[ − 1 , 1] d , and K = { x ∈ R d | ∥ x ∥ 2 ≤ C } .
Let µ n be the n -sample empirical measure corresponding to a probability measure µ over [ − 1 , 1] d .
When µ is the uniform probability measure and n ≥ C 1 log( d ) , we have that
[Equation]
For any probability measure µ over [ − 1 , 1] d ,
[Equation]
where K is a constant depending on C and d (but not the measure µ ).
Overlooking logarithmic factors, we can summarize Theorem 5 as E [ d CT , A ( µ, µ n )]
≍ n − 2 d .
The estimation of the CT distance is cursed by dimension: one needs a sample size exponential with the dimension d for µ n to be at a desired distance from µ .
It is also interesting to contrast the rate n − 2 d with the rates for similar distances.
For example, for the r -Wasserstein distance we have E [ W rr ( µ, µ n )] ≍ n − rd (Singh & P ´ oczos (2018), Section 4).
Given the link of the CT distance and
VDC with the squared 2-Wasserstein distance (see Subsec. 3.1), the n − 2 d rate is natural.
The proof of Theorem 5, which can be found in App.
D, is based on upper-bounding and lowerbounding the metric entropy of the class of bounded Lipschitz convex functions with respect to an appropriate pseudo-norm.
Then we use Dudley’s integral bound and Sudakov minoration to upper and lower-bound the Rademacher complexity of this class, and we finally show upper and lower bounds of the CT distance by the Rademacher complexity.
Bounds on the metric entropy of bounded
Lipschitz convex functions have been computed and used before (Balazs et al., 2015; Guntuboyina &
Sen, 2013), but in L p and supremum norms, not in our pseudo-norm.
Next, we see that the surrogate CT distance defined in (6) does enjoy parametric estimation rates.
Theorem 6.
Suppose that for all 1 ≤ ℓ ≤ L , the widths m ℓ satisfy m ℓ ≤ m .
We have that
[Equation]
In short, we have that E [ d CT ,F L, M ,k, + (1) ( µ, µ n )]
= O ( Lm (cid:113) kn ) .
Hence, if we take the number of samples n larger than k times the squared product of width and depth, we can make the surrogate
CT distance small.
Theorem 6, proven in App.
D, is based on a Rademacher complexity bound for the space F L, M ,k (1) of maxout networks which may be of independent interest; to our knowledge, existing Rademacher complexity bounds for maxout networks are restricted to depth-two networks (Balazs et al., 2015; Kontorovich, 2018).
Theorems 5 and 6 show the advantages of the surrogate CT distance over the CT distance are not only computational but also statistical; the CT distance is such a strong metric that moderate-size empirical versions of a distribution are always very far from it.
Hence, it is not a good criterion to compare how close an empirical distribution is to a population distribution.
In contrast, the surrogate
CT distance between a distribution and its empirical version is small for samples of moderate size.
An analogous observation for the Wasserstein distance versus the neural net distance was made by
Arora et al.
If µ n , ν n are empirical versions of µ, ν , it is also interesting to bound | d CT ,F L, M ,k, + (1) ( µ n , ν n ) − d CT , A ( µ, ν ) | ≤ | d CT ,F L, M ,k, + (1) ( µ n , ν n ) − d CT ,F L, M ,k, + (1) ( µ, ν ) | + | d CT ,F L, M ,k, + (1) ( µ, ν ) − d CT , A ( µ, ν ) | .
The first term has a O ( (cid:112) k/n ) bound following from Theorem 6, while the second term is upper-bounded by 2 sup f ∈A inf ˜ f ∈ F L, M ,k (1) ∥ f − ˜ f ∥ ∞ , which is (twice) the approximation error of the class A by the class F L, M ,k (1) .
Such bounds have only been derived in L = 2 : Balazs et al.
(2015) shows that sup f ∈A inf ˜ f ∈ F 2 , ( d, 1) ,k (1) ∥ f − ˜ f
∥ ∞ =
O ( dk − 2 /d ) .
Hence, we need k exponential in d to make the second term small, and thus n ≫ k exponential in d to make the first term small.
L EARNING DISTRIBUTIONS WITH THE SURROGATE VDC AND CT DISTANCE
We provide a min-max framework to learn distributions with stochastic orders.
As in the generative adversarial network (GAN, Goodfellow et al.
Arjovsky et al.
(2017)) framework, we parametrize probability measures implicitly as the pushforward µ = g # µ 0 of a base measure µ 0 by a generator function g in a parametric class G and we optimize over g .
The loss functions involve a maximization over ICMNs corresponding to the computation of a surrogate VDC or CT distance (and possibly additional maximization problems), yielding a min-max problem analogous to GANs.
Enforcing dominance constraints with the surrogate VDC.
In some applications, we want to optimize a loss L : P (Ω) → R under the constraint that µ = g # µ 0 dominates a baseline measure ν .
We can enforce, or at least, bias µ towards the dominance constraint by adding a penalization term proportional to the surrogate VDC between µ and ν , in application of Proposition 2.
A first instance of this approach appears in portfolio optimization (Xue et al., 2020; Post et al., 2018).
Let ξ =
( ξ 1 , . .
. , ξ p ) be a random vector of return rates of p assets and let Y 1 :
= G 1 ( ξ ) , Y 2 :
= G 2 ( ξ ) be real-valued functions of ξ which represent the return rates of two different asset allocations or portfolios, e.g. G i ( ξ ) = ⟨ ω i , ξ ⟩ with ω i ∈
R p .
The goal is to find a portfolio G 2 that enhances a benchmark portfolio G 1 in a certain way.
For a portfolio G with return rate Y := G ( ξ ) , we let
F (1) Y ( x ) =
P ξ ( Y ≤ x ) be the CDF of its return rate, and F (2) Y ( x )
= (cid:82) x
−∞ F (1) Y ( x ′ )
dx ′ .
If Y 1 , Y 2 are the return rates of G 1 , G 2 , we say that Y 2 dominates Y 1 in second order, or Y 2 ⪰ 2 Y 1 if for all x ∈ R ,
F (2) Y 2 ( x ) ≤ F (2)
Y 1 ( x ) , which intuitively means that the return rates Y 2 are less spread out than those Y 1 , i.e. the risk is smaller.
Formally, the portfolio optimization problem can be written as:
[Equation]
It turns out that Y 2 ⪰ 2 Y 1 if and only if E [( η − Y 2 ) + ] ≤ E [( η − Y 1 )
+ ] for any η ∈ R , or yet equivalently, if E [ u ( Y 2 )] ≥ E [ u ( Y 1 )] for all concave non-decreasing u : R → R (Dentcheva &
Ruszczy ´ nski, 2004).
Although different, note that the second order is intimately connected to the
Choquet order for 1-dimensional distributions, and it can be handled with similar tools.
Define
F L, M ,k, − + (1) as the subset of F L, M ,k, + (1) such that the first m 1 components of the weights w (1) i,j are non-positive for all 1 ≤ i ≤ m 2 , 1 ≤ j ≤ k .
If we set the input width m 1 = 1 , we can encode the condition Y 2 ⪰ 2 Y 1 as VDC F L, M ,k, − + (1) ( ν || µ ) , where ν
=
L ( Y 1 ) and µ =
L ( Y 2 ) are the distributions of Y 1 , Y 2 , resp.
Hence, with the appropriate Lagrange multiplier we convert problem (9) into a min-max problem between µ and the potential u of VDC
[Equation]
A second instance of this approach is in GAN training.
Assuming that we have a baseline generator g 0 that can be obtained via regular training, we consider the problem:
[Equation]
The first term in the objective function is the usual WGAN loss (Arjovsky et al., 2017), although it can be replaced by any other standard GAN loss.
The second term, which is proportional to
[Equation]
u
[Equation]
[Equation]
[Equation]
enforces that g # µ 0 ⪰ ( g 0 ) # µ 0 in the Choquet order, and thus u acts as a second ‘Choquet’ critic.
Tuning λ appropriately, the rationale is that we want a generator that optimizes the standard GAN loss, with the condition that the generated distribution dominates the baseline distribution.
As stated by Proposition 1, dominance in the Choquet order translates to g # µ 0 being more spread out than ( g 0 )
# µ 0 , which should help avoid mode collapse and improve the diversity of generated samples.
In practice, this min-max game is solved via Algorithm 1 given in App.
F.
For the Choquet critic, this amounts to an SGD step followed by a projection step to impose non-negativity of hidden to hidden weights.
Generative modeling with the surrogate CT distance.
The surrogate Choquet-Toland distance is well-suited for generative modeling, as it can used in GANs in place of the usual discriminator.
Namely, if ν is a target distribution, ν n is its empirical distribution, and D = { µ = g
# µ 0 | g ∈ G} is a class of distributions that can be realized as the push-forward of a base measure µ 0 ∈ P ( R d 0 ) by a function g : R d 0 → R d , the problem to solve is
[Equation]
Algorithm 2 given in App.
F summarizes learning with the surrogate CT distance.
E XPERIMENTS
Portfolio optimization under dominance constraints In this experiment, we use the VDC to optimize an illustrative example from Xue et al.
(Example 1) that follows the paradigm laid out in Sec. 6.
In this example, ξ ∈ R ∼ P is drawn uniformly from [0 , 1] , we define the benchmark portfolio as:
[Equation]
and the optimization is over the parameterized portfolio G 2 ( ξ ) :
= G ( ξ ; z ) =
zξ .
The constrained optimization problem is thus specified as:
[Equation]
As stated in Xue et al.
(2020), this example has a known solution at z = 2 where E P [ G 2 ( ξ ; 2)]
=outperforms the benchmark E P [ G 1 ( ξ )]
= 0 .
We relax the constrained optimization by including it in the objective function, thus creating min-max game (10) introduced in Sec.
We parameterize
F L, M ,k, − + (1) with a 3-layer, fully-connected, decreasing ICMN with hidden dimension 32 and maxout kernel size of 4.
After 5000 steps of stochastic gradient descent on z (learning rate 1 e − 3 ) and the parameters of the ICMN (learning rate 1 e − 3 ), using a batch size of 512 and λ = 1 , we are able to attain accurate approximate values of the known solution:
z = 2 , 1512 (cid:80)
j =1 [ G z ( ξ j )]
= 1 . 042 , and 1512 (cid:80)
j =1 [ G 1 ( ξ j )]
= 0 .
[Figure]
Image generation with baseline model dominance Another application of learning with the VDC is in the high-dimensional setting of CIFAR-10 (Krizhevsky & Hinton, 2009) image generation.
As detailed in Sec. 6, we start by training a baseline generator g 0 using the regularized WassersteinGAN paradigm (WGAN-GP) introduced in Arjovsky et al.
Gulrajani et al.
(2017), where gradient regularization is computed for the discriminator with respect to interpolates between real and generated data.
When training g ∗ and g 0 , we used the same WGAN-GP hyperparameter configuration.
We set λ in Equation (11) to 10 (see App.
F and App.
H for more details).
Training runs for g ∗ and g
[Table]
were performed in computational environments that contained 1 CPU and 1 A100 GPU.
To evaluate performance of g ∗ vs. g 0 , we rely on the Fr´echet Inception Distance (FID) introduced in Heusel et al.
Note that when training a WGAN-GP baseline from scratch we used hyperparameters that potentially differ from those used in state-of-the-art implementations.
Additionally, for computing
FIDs we use a pytorch-lightning implementation of the inception network, which is different from the widely used Tensorflow implementation (Salimans et al., 2016), resulting in potential discrepancies in our reported baseline FID and those in the literature.
FID results are reported in
Table 1, where we see improved image quality from g ∗ (as measured by lower FID) relative to the pre-trained baseline g 0 .
We therefore find that the VDC surrogate improves upon g 0 by providing g ∗ with larger support, preventing mode collapse.
Samples generated from g ∗ are displayed in Figure 2a.
In order to highlight the representation power of ICMNs, we replace them in the VDC estimation by the ICNN implementation of Huang et al.
Instead of maxout activation, Huang et al.
(2021) uses a Softplus activation and instead of a projection step it also uses a Softplus operation to impose the non-negativity of hidden to hidden weights to enforce convexity.
We see in Table 1 that VDC’s estimation with ICMN outperforms Huang et al.
(2021) ICNNs.
While ICMNs have maximum of affine functions as a building block, ICNN’s proof of universality in Huang et al.
(2021) relies on approximating the latter, this could be one of the reason behind ICMN superiority.
Probing mode collapse To investigate how training with the surrogate VDC regularizer helps alleviate mode collapse in GAN training, we implemented GANs trained with the IPM objective alone and compared this to training with the surrogate VDC regularizer for a mixture of 8 Gaussians target distribution.
In Figure 3 given in App.
G we quantify mode collapse by looking at two scores: 1) the entropy of the discrete assignment of generated points to the means of the mixture 2) the negative log likelihood (NLL) of the Gaussian mixture.
When training with the VDC regularizer to improve upon the collapsed generator g 0 (which is taken from step 55k from the unregularized GAN training), we see more stable training and better mode coverage as quantified by our scores.
2D point cloud generation with d CT We apply learning with the CT distance in a 2D generative modeling setting.
Both the generator and CT critic architectures are comprised of fully-connected neural networks with maxout non-linearities of kernel size 2.
Progression of the generated samples can be found in the right-hand panel of Figure 2, where we see the trained generator accurately learn the ground truth distribution.
All experiments were performed in a single-CPU compute environment.
C ONCLUSION
In this paper, we introduced learning with stochastic order in high dimensions via surrogate Variational
Dominance Criterion and Choquet-Toland distance.
These surrogates leverage input convex maxout networks, a new variant of input convex neural networks.
Our surrogates have parametric statistical rates and lead to new learning paradigms by incorporating dominance constraints that improve upon a baseline.
Experiments on synthetic and real image generation yield promising results.
Finally, our work, although theoretical in nature, can be subject to misuse, similar to any generative method.
G Probing mode collapse
H Additional experimental details
I
Assets A P ROOFS OF S EC .
Proof of Proposition 2.
Since A is included in the set of convex functions, the implication from right to left is straightforward.
For the other implication, we prove the contrapositive.
Suppose that u is a convex function on Ω such that (cid:82) Ω u d ( µ − − µ + ) > 0 .
Then we show that we can construct ˜ u
∈
A such that (cid:82) Ω ˜ u d ( µ − − µ + ) > 0 , which shows that the supremum over A is strictly positive.
Denote by C the set of convex functions f : R d → R which are the point-wise supremum of finitely many affine functions, i.e. f ( x ) = max
i ∈
I {⟨ y i , x ⟩ − a i } for some finite family ( y i , a i ) ∈ R d × R .
For any convex function g , there is an increasing sequence ( g n ) n ⊆ C such that g
= sup n g n pointwise (Ekeland & Schachermayer (2014), p. 3).
Applying this to u , we know there exists an increasing sequence ( u n ) n ⊆ C such that u = lim n →∞ u n .
By the dominated convergence theorem,
[Equation]
which means that for some N large enough, (cid:82) Ω u N d ( µ − − µ + ) > 0 as well.
Since u N admits a representation f ( x ) = max i ∈ I {⟨ y i , x ⟩ − a i } for some finite family I
, it may be trivially extended to a convex function on R d .
Let ( η ϵ )
ϵ be a family of non-negative radially symmetric functions in C 2 ( R d ) supported on the ball
B ϵ (0) of radius ϵ centered at zero, and such that (cid:82) R d η ϵ ( x ) dx = 1 .
Let Ω
ϵ = { x ∈ Ω | dist ( x, ∂ Ω) ≥ ϵ } .
For any x ∈ R d , we have that
[Equation]
[Equation]
By the convexity of u N , we have that u N ( λx +(1 − λ ) x ′ − y ′ )
≤ λu N ( x − y ′ )
+(1 − λ ) u N ( x ′ − y ′ ) for any x, x ′ , y ′ ∈ R d .
Thus, (12) implies that ( u N ∗ η ϵ )( λx + (1 − λ ) x ′ ) ≤ λ ( u N ∗ η ϵ )( x ) + (1 − λ )
( u N ∗ η ϵ )( x ′ ) , which means that u N ∗ η ϵ is convex.
Also, by the dominated convergence theorem,
[Equation]
Hence, there exists ϵ 0 > 0 such that (cid:82) Ω ( u N ∗ η ϵ ) d ( µ − − µ + ) > 0 .
Since u N ∗ η ϵ 0 is in C 2 ( R d ) by the properties of convolutions, its gradient is continuous.
Since the closure ¯Ω is compact because Ω is bounded, by the Weierstrass theorem we have that
[Equation]
Let r > 0 be such that the ball B r (0) is included in K .
Rescaling u N ∗ η ϵ 0 by an appropriate constant, we have that sup x ∈ Ω ∥∇ ( u N ∗ η ϵ 0 )( x ) ∥ 2 < r , and that means that ∇ ( u N ∗ η ϵ 0 )( x ) ∈ K for any x ∈ Ω .
Thus, u N ∗ η ϵ 0 ∈ A , and that means that sup u ∈A { (cid:82) Ω u d ( µ − − µ + ) } > 0 , concluding the proof.
Lemma 1.
Let Ω =
[ − 1 , 1] d
and K = { x ∈ R d | ∥ x ∥ 2 ≤ C } .
The function class A = { u : Ω → R , u convex and ∇ u a.e. ∈ K} is equal to the space F d ( M, C ) of convex functions on [ − 1 , 1] d such that | f ( x ) | ≤ M and | f
( x ) − f ( y ) | ≤ C ∥ x − y ∥ for any x, y ∈ [ − 1 , 1] d , up to a constant term.
Here, M = C √ d .
Proof.
Looking at problem (1), note that adding a constant to a function u ∈ A does not change the value of the objective function.
Thus, we can add the restriction that u (0) = 0 , and since Ω is compact and has Lipschitz constant upper-bounded by sup x ∈ K ∥ x ∥ , any such function u must fulfill M := sup u ∈A sup x ∈ Ω | u ( x ) | < + ∞ .
Thus, we have that functions in A belong to { u : Ω → R , u convex and ∇ u a.e. ∈ K, sup x ∈ Ω | u ( x ) | ≤ M } up to a constant term, for a well chosen M .
Now we will use the particular form of Ω and K .
First, note that we can take M = C √ d without loss of generality.
Given u ∈ A , we have that ∥∇ u ( x ) ∥ 2 ≤ C for a.e. x ∈ [ − 1 , 1] d .
By the mean value theorem, we have that | u ( x ) − u ( y ) | = | (cid:82) 1 0 ⟨∇ f ( tx + (1 − t ) y ) , x − y ⟩ dt | ≤ C ∥ x − y ∥ , implying that u is C -Lipschitz.
This shows that A ⊆ F d ( M, C ) .
Rademacher’s theorem states that C -Lipschitz functions are a.e. differentiable, and gradient norms must be upper-bounded by C wherever gradients exist as otherwise one reaches a contradiction.
Hence, F d ( M, C ) ⊆
A , concluding the proof.
Lemma 2 (Metric entropy of convex functions, Bronshtein (1976), Thm. 6) .
Let F d ( M, C ) be the compact space of convex functions on [ − 1 , 1] d such that | f ( x ) | ≤ M and | f
( x ) − f ( y ) | ≤ C ∥ x − y ∥ for any x, y ∈ [ − 1 , 1] d .
The metric entropy of this space with respect to the uniform norm topology satisfies
[Equation]
for some constant K that depend on C , M and d .
Lemma 3 (Dudley’s entropy integral bound, Wainwright (2019), Thm.
5.22, Dudley (1967)) .
Let { X θ | θ ∈ T } be a zero-mean sub-Gaussian process with respect the metric ρ X on T .
Let
[Equation]
ρ X ( θ, θ ′ ) .
Then for any δ ∈ [0 , D ] such that N ( δ ; T , ρ X ) ≥ 10 , we have
[Equation]
Proposition 4.
For any family of n points ( X i ) ni =1 ⊆
[ − 1 , 1] d , the empirical Rademacher complexity of the function class F d ( M, C ) satisfies
[Equation]
where K is a constant depending on M , C and d .
Proof.
We choose T n = { ( f ( X i ))
ni =1 ∈ R n | f ∈ F d ( M, C ) } , we define the Rademacher process X
f = (cid:80) n i =1 ϵ i f
( X i ) , which is sub-Gaussian with respect to the metric ρ n [Equation] f, f
[Equation] ) = [Equation] ni =
[-=
-][Equation]
f ( X i ) − f ′ ( X i )) 2 .
Remark that D ≤ 2 M √ n .
For any δ ∈ [0 , D ] , we apply Lemmasetting f ′ ≡
and we get
[Equation]
Note that for any f, f ′ ∈ F d ( M, C ) , ρ n ( f, f ′ ) ≤ √ n ∥ f − f ′ ∥ ∞ , which means that log N ( δ ; F d ( M, C ) , ρ n ) ≤ log N ( δ [Equation] √ n ; F d ( M, C ) , ∥ · ∥ ∞ ) .
Thus, Lemma 2 implies that
[Equation]
Hence,
[Equation]
We set δ = n 12 − 2 d , and we get that δ − d 4 +1 = n ( 12 − 2 d )( − d 4 +1)
=
n − d 8 +1 − 2 d .
Hence, the right-hand side of (14) is upper-bounded by √ K d 4 +1 n 1 − 2 d .
And since (cid:80) ni
=1 ϵ
i ( f ( X i ) − f ′ ( X i )) ≤ (cid:80) ni =1 | f ( X i ) − f ′ ( X i ) | ≤ √ nρ n ( f, f ′ ) , we have
[Equation]
Plugging these bounds back into (13), we obtain
[Equation]
Since K already depends on d , we rename it as K ← √ K d 4 +1 + 1 , concluding the proof.
Proof of Theorem 2.
Let F d ( C ) be the space of convex functions on [ − 1 , 1] d such that | f ( x ) | ≤ C √ d and | f ( x ) − f ( y ) | ≤ C ∥ x − y ∥ for any x, y ∈ [ − 1 , 1] d .
Lemma 1 shows that when Ω = [ − 1 , 1] d and
K = { x ∈ R d | ∥ x ∥ 2 ≤ C } , functions in A belong to F d ( C ) up to a constant term, which means that
VDC [Equation] µ || µ ) =
sup u ∈ F d ( C ) (cid:82) Ω u d ( µ − − µ + ) .
Theorem 11 of Sriperumbudur et al.
(2009) shows that for any function class F on Ω such that r := sup f ∈F ,x ∈ Ω | f ( x ) | < + ∞ , with probability at least 1 − δ
we have
[Equation]
where ˆ R n denotes the empirical Rademacher complexity.
Proposition 4 shows that ˆ
R n ( F d ( C ) , ( x i ) ni
=1 ) ≤ K n − 2 d for any ( x i ) ni =1 ∈ Ω ⊆
[ − 1 , 1] d , where K depends on C and d .
This concludes the proof.
B P ROOFS OF S EC .
Proposition 5.
Input convex maxout networks (Definition 3) are convex with respect to their input.
Proof.
The proof is by finite induction.
We show that for any 2 ≤ ℓ ≤ L − 1 and 1 ≤ i ≤ m ℓ , the function x
(cid:55)→ x ( ℓ ) i is convex.
The base case ℓ = 2 holds because x (cid:55)→ x (2)
i
[Equation]
is a pointwise supremum of convex (affine) functions, which is convex.
For the induction case, we have that x (cid:55)→ x ( ℓ − 1) i is convex for any 1 ≤ i ≤ m ℓ − 1 by the induction hypothesis.
Since a linear combination of convex functions with non-negative coefficients is convex, we have that for any 1 ≤ i ≤ m ℓ , 1 ≤ j ≤ k , x
(cid:55)→ ⟨ w ( ℓ − 1) i,j , ( x ( ℓ − 1) , 1) ⟩ is convex.
Finally, x (cid:55)→ x ( ℓ ) i =
[Equation]
) ⟩ is convex because it is the pointwise supremum of convex functions.
Proof of Proposition 3.
We can reexpress f ( x ) as:
[Equation]
For 1 ≤ ℓ ≤ L − 1 , we define the matrices W ∗ ℓ ∈ R m ℓ +1 ,m ℓ such that their i -th row is the vector [ 1 √ m ℓ +1 w ( ℓ ) i,j ∗ ℓ,i ] 1: m ℓ , i.e. the vector containing the first m ℓ components of 1 √ m
[Equation]
+1 w ( ℓ ) i,j ∗ ℓ,i .
Iterating the chain rule, one can see that for almost every x ∈ R d ,
[Equation]
Since the spectral norm ∥ · ∥ 2 is sub-multiplicative and ∥ A ∥ 2 = ∥
A ⊤ ∥ 2 , we have that ∥∇ f
( x ) ∥ =
∥ ( W ∗ 1 ) ⊤ ( W ∗ 2 ) ⊤ . . .
( W ∗ L − 1 ) ⊤ ∥ 2 ∥ a ∥ =
∥ W ∗ 1 ∥ 2 ∥ W ∗ 2 ∥ 2 . . .
∥ W ∗ L − 1 ∥ 2 ∥ a ∥ .
We compute the Frobenius norm of W ∗ ℓ :
[Equation]
Since for any matrix A , ∥ A ∥ 2 ≤ ∥ A ∥ F , and the vector a satisfies ∥ a ∥ ≤ 1 , we obtain that ∥∇ f ( x ) ∥ ≤ 1 .
To obtain the bound on | f ( x ) | , we use again the expression (15).
For 1 ≤ ℓ ≤ L − 1 , we let b ℓ ∈ R m ℓ +1 be the vector such that the i -th component is [ 1 √ m ℓ +1 w ( ℓ ) i,j ∗ ℓ,i ] m ℓ +1 .
Since | [ 1 √ m ℓ +1 w ( ℓ ) i,j ∗ ℓ,i ] m ℓ +1 | ≤
[Equation]
, ∥ b ℓ ∥ ≤ 1 .
It is easy to see that
[Equation]
Thus, | f ( x ) | ≤ ∥ a ∥
[Equation]
≤
[Equation]
The bound on ∥ x ( ℓ ) ∥ follows similarly, as x ℓ =
W ∗ ℓ − 1 · · · W ∗ 1 x + (cid:80) ℓ − 1 ℓ ′ =1 a ⊤ W ∗ ℓ − 1 · · · W ∗ ℓ +1 b ℓ .
Proposition 6.
Let F L, M ,k (1) be the subset of F L, M ,k such that for all 1 ≤ ℓ ≤ L − 1 , 1 ≤ i ≤ m ℓ , 1 ≤ j ≤ k , ∥ w ( ℓ ) i,j ∥ 2 ≤ 1 , and ∥ a ∥ 2 = (cid:80) m
L i =1 a 2 i ≤ 1 .
For any f ∈ F L, M ,k (1) and x ∈ B 1 ( R d ) , we have that | f ( x ) | ≤ 1 .
The metric entropy of F L, M ,k (1) with respect to ˜ ρ n admits the upper bound: log N (cid:0) δ ; F L, M ,k (1) , ˜ ρ n (cid:1) ≤ (cid:80) Lℓ =2 km ℓ ( m ℓ − 1 + 1) log (cid:16) 1 + 2 3+ L − ℓ δ
[Equation]
Proof.
We define the function class G L, M ,k that contains the functions from R d to R m L of the form g
[Equation]
Given { X i } ni =1 ⊆
B 1 ( R d ) we define the pseudo-metric ˜ ρ n between functions from R d to R m L as
[Equation]
We prove by induction that
[Equation]
To show the induction case, note that for L ≥ 3 , any g ∈ G L, M ,k can be written as
[Equation]
where h ∈ G L − 1 , M ,k .
Remark that given a δ 2 √
( L − 1) 2 +1 -cover C ′ of B 1 ( R m L − 1 +1 ) , there exist ˜ w
( L − 1) i,j ∈ C ′ such that ∥ ˜ w ( L − 1) i,j − w ( L − 1) i,j ∥ ≤ δ 2 √
( L − 1) 2 +1 .
Hence, if ˜ h is such that ˜ ρ n ( ˜ h, h ) ≤ δ 2 , and we define ˜ g ( x )
=
( 1 √ 2 m L max j ∈ [ k ] ⟨ ˜ w ( L − 1) i,j , (˜ h ( x ) , 1) ⟩ ) m
L i =1 , we obtain ˜ ρ n (˜ g, g )=
1 nm L (cid:80) m
L i =1 (cid:80)
nk = (cid:18) max j ∈ [ k ] ⟨ ˜ w ( L − 1) i,j , (˜ h
( X k ) , 1) ⟩ − max j ∈ [ k ] ⟨ w ( L − 1) i,j , ( h ( X k ) , 1) ⟩ (cid:19)≤ 1 nm L (cid:80) m
L i =1 (cid:80) nk =1 max j ∈ [ k ] (cid:18) ⟨ ˜ w ( L − 1) i,j − w ( L − 1) i,j , (˜ h ( X k ) , 1) ⟩ − ⟨ w ( L − 1) i,j , ( h ( X k ) − ˜ h
( X k ) , 0) ⟩ (cid:19)≤ 2 nm L (cid:80) m
L i =1 (cid:80) nk =1 max j ∈ [ k ] (cid:18) ⟨ ˜ w ( L − 1) i,j − w ( L − 1) i,j , (˜ h ( X k ) , 1) ⟩ (cid:19) + (cid:18) ⟨ w ( L − 1) i,j , ( h ( X k ) − ˜ h
( X k ) , 0) ⟩ (cid:19)≤ 2 nm L (cid:80) m
L i =1 (cid:80) nk =1 max j ∈ [ k ] ∥ ˜ w ( L − 1) i,j − w ( L − 1) i,j ∥ 2 ∥ (˜ h ( X k ) , 1) ∥ 2 + ∥ w ( L − 1) i,j ∥ 2 ∥ h ( X k ) − ˜ h
( X k ) ∥≤ 2 nm L (cid:80)
[Equation]
In the second-to-last inequality we used that
if h ∈ G ℓ, M ,k , ∥ h ( x ) ∥ ≤ ℓ .
This is equivalent to the bound ∥ x ( ℓ ) ∥ ≤ ℓ shown in Proposition 3.
Hence, we can build a δ -cover of G L, M ,k in the pseudo-metric ˜ ρ n from the Cartesian product of a δ 2 -cover of G L − 1 , M ,k in ˜ ρ n and km L copies of a δ 2 √ ( L − 1) 2 +1 -cover of B 1 ( R m L − 1 +1 ) in the ∥ · ∥ 2 norm.
Thus,
[Equation]
The metric entropy of the unit ball admits the upper bound log N ( δ ; B 1 ( R d ) , ∥ ·
∥ 2 ) ≤ d log(1 + 2 δ ) (Wainwright (2019), Example 5.8).
Consequently, log N ( δ ; G L, M
[Equation]
In the second inequality we used the induction hypothesis
[Equation]
To conclude the proof
[Equation]
note that an arbitrary function f ∈ F L, M ,k (1) can be written as f ( x ) = ⟨ a, g ( x ) ⟩ , where a ∈ B 1 ( R m L ) and g ∈ G L, M ,k .
Applying an analogous argument, we see that a δ 2 √ L 2 +1 -cover of B 1 ( R m L ) and a δ 2 -cover of G L, M ,k give rise to a δ -cover of F L, M ,k (1) .
Hence,
[Equation]
(cid:17) .
Finally, to show the bound | f ( x ) | ≤ 1 for all f ∈ F L, M ,k (1) and x ∈ B 1 ( R d ) , we use that | f ( x ) | ≤ |⟨ a, g ( x ) ⟩| ≤ ∥ a ∥∥ g ( x ) ∥ ≤ 1 and that if g ∈ G ℓ, M ,k and x ∈ B 1 ( R d ) , then ∥ g ( x ) ∥ ≤ 1 , as shown before.
Proposition 7.
Suppose that for all 1 ≤ ℓ ≤ L , the widths m ℓ satisfy m ℓ ≤ m .
Then, the Rademacher complexity of the class F L, M ,k (1) satisfies:
[Equation]
Proof.
We apply Dudley’s entropy integral bound (Lemma 3).
We choose T n = { ( f ( X i ))
ni =1 ∈ R n | f ∈ F L, M ,k (1) } , we define the Rademacher process X f = 1 √ n (cid:80) ni
=1 ϵ
i f ( X i ) , which is subGaussian with respect to the metric ˜ ρ n ( f, f ′ ) =
(cid:113) 1 n (cid:80) ni
=
( f ( X i ) − f ′ ( X i )) 2 .
Remark that D ≤ 2 .
Setting f ′ ≡ 0 and δ = 0 in Lemma 3, we obtain that
[Equation]
Applying the metric entropy bound from Proposition 6, we get
[Equation]
In the last equality we used that for δ ∈ [0 , 2] [Equation] 1 + 2 L +1 √ L 2 +1 δ ≤ 2 L [-+
-]+2 √ L 2 +1 δ .
We compute the integral
[Equation]
Applying Lemma 4 with z = 2 − ( L +1)
[Equation]
L 2 +
[Equation]
L 2 +
[Equation]
we obtain that
[Equation]
Putting everything together yields equation (16)
[Equation]
Lemma 4.
For any z ∈ (0 , 1]
[Equation]
we have that
[Equation]
Here, erfc denotes the complementary error function, defined as erfc( x ) = 2 √ π (cid:82) + ∞ x
[Equation]
Proof.
We rewrite the integral as:
[Equation]
The complementary error function satisfies the bound erfc( x ) ≤ e − x 2 for any x > 0 , which implies the final inequality.
Proof of Theorem 3.
Proposition 7 proves that under the condition m ℓ ≤ m , the empirical Rademacher complexity of the class F L, M ,k (1) satisfies ˆ R n ( F L, M ,k (1)) ≤ 64 (cid:113)
( L − 1) km ( m +1) n ( (cid:113) ( L + 1) log(2)
+ 12 log( L 2 + 1) + √ π 2 ) .
Applying Theorem 11 of Sriperumbudur et al.
(2009) as in the proof of Theorem 2, and that for any f ∈ F L, M ,k (1) and x ∈ B r ( R d ) , we have that | f
( x ) | ≤ r (see Proposition 6), we obtain the result.
C P ROOFS OF S EC .
Proof of Theorem 4.
To show (i), we have that for any ν
′ ∈
P
( K ) ,
[Equation]
The right-to-left implication of (ii) is straight-forward.
To show the left-to-right one, we use the definition for the CT distance, rewriting VDC A ( µ + || µ − ) and VDC A ( µ − || µ + ) in terms of their definitions:
[Equation]
(cid:27) .
Since the two terms in the right-hand side are non-negative, d CT , A ( µ + , µ − )
= 0 implies that they are both zero.
Then, applying Proposition 2, we obtain that µ − ⪯ µ + and µ + ⪯ µ − according to the
Choquet order.
The antisymmetry property of partial orders then implies that µ + = µ − .
To show (iii), we use equation (17) again.
The result follows from sup u ∈A (cid:26) (cid:82) [Equation] (cid:82)
[Equation]
(cid:27)
[Equation]
sup [Equation] (cid:26) (cid:82) [Equation]
(cid:27)
[Equation]
(cid:26) (cid:82)
[Equation]
(cid:27) + [Equation] u
[Equation]
(cid:26) (cid:82)
[Equation]
(cid:27) .
D P ROOFS OF S UBSEC . 5.
Lemma 5.
For a function class F that contains the zero function, define ∥ S n ∥ F = sup f ∈F | 1 n (cid:80) ni
=1 ϵ
i f ( X i ) | , where ϵ i are Rademacher variables.
E X,ϵ
[ ∥ S n ∥ F ] is known as the
Rademacher complexity.
Suppose that 0 belongs to the compact set K .
We have that
[Equation]
where ¯ A = { f − E µ [ f ] | f ∈ A} is the centered version of the class A .
Proof.
We will use an argument similar to the proof of Prop. 4.11 of Wainwright (2019) (with the appropriate modifications) to obtain the Rademacher complexity upper and lower bounds.
We start
[Table]
[Equation]
The last inequality follows from the fact that sup f ∈A | 1 n (cid:80) ni =1 ( f ( X i ) − E
[ f ]) | ≤ sup f ∈A 1 n (cid:80) ni
=
( f ( X i ) − E
[ f ]) + sup f ∈A 1 n (cid:80) ni
=
( E [ f ] − f ( X i )) , which holds as long as the two terms in the right-hand side are non-negative.
This happens when f ≡ 0 belongs to A as a consequence of 0 ∈ K .
The upper bound follows essentially from the classical symmetrization argument:
[Equation]
Lemma 6 (Relation between Rademacher and Gaussian complexities, Exercise 5.5, Wainwright (2019)) .
Let ∥ G n ∥ F = sup f ∈F | 1 n (cid:80) ni
=
z i f ( X i ) | , where z i are standard Gaussian variables.
E
X,z
[ ∥ G n ∥ F ] is known as the Gaussian complexity.
We have
[Equation]
Given a set T ⊆ R n , the family of random variables { G θ , θ ∈ T } , where G θ := ⟨ ω, θ ⟩ = (cid:80) ni =1 ω i θ i and ω i ∼ N (0 , 1) i.i.d., defines a stochastic process is known as the canonical Gaussian process associated with T .
D. R ESULTS USED IN THE LOWER BOUND OF T HEOREM
Lemma 7 (Sudakov minoration, Wainwright (2019), Thm. 5.30; Sudakov (1973)) .
Let { G θ , θ ∈ T } be a zero-mean Gaussian process defined on the non-empty set T .
Then,
[Equation]
where M G ( δ ; T ) is the δ -packing number of T in the metric ρ G ( θ, θ ′ )
= (cid:112) E
[( X θ − X θ ′ ) 2 ] .
Proposition 8.
Let C 0 , C 1 be universal constants independent of the dimension d , and suppose that n ≥ C 1 log( d ) .
Recall that F d ( M, C ) is the set of convex functions on [ − 1 , 1] d such that | f ( x ) | ≤ M and | f
( x ) − f ( y ) | ≤ C ∥ x − y ∥ for any x, y ∈ [ − 1 , 1] d , and that ¯ F d
[Equation]
M, C ) =
{ f − E µ [ f ] | f ∈
F d ( M, C ) } .
The Gaussian complexity of the set F d ( M, C ) satisfies
[Equation]
Proof.
Given ( X i ) ni
=1 ⊆ R d , let T n = { ( f ( X i ))
ni =1 ∈ R n | f ∈ F d ( M, C ) } .
Let X i be sampled i.i.d.
from the uniform measure on [ − 1 , 1] d .
We have that with probability at least 1 / 2 on the instantiation of ( X i ) ni =1 ,
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
The first inequality follows from Sudakov minoration (Lemma 7) by setting δ = (cid:113)
[Equation]
The second inequality follows from the lower bound on the packing number of ¯ A given by Corollary 1.
In the following approximation we neglected the term 1 in the numerator inside of the logarithm.
In the last inequality, C 0 is a universal constant independent of the dimension d .
The inequality holds as long as log(32 (cid:112)
2 d (1 + C ) C )
+ ( 12 + 2 d ) log( n ) ≤ n 32 log(2) , which is true when n ≥ C 1 log( d ) for some universal constant C 1 .
Proposition 9.
With probability at least (around) 1 / 2 on the instantiation of ( X i ) ni
=1 as i.i.d.
samples from the uniform distribution on S d − 1 , the packing number of the set F d ( M, C ) with respect to the metric ρ n ( f, f ′ ) =
(cid:112)(cid:80) ni
=
( f ( X i ) − f ′ ( X i ))
2 satisfies
[Equation]
Proof.
This proof uses the same construction of Thm.
6 of Bronshtein (1976), which shows a lower bound on the metric entropy of F d ( M, C ) in the L ∞ norm.
His result follows from associating a subset of convex functions in F d ( M, C ) to a subset of convex polyhedrons, and then lower-bounding the metric entropy of this second set.
Note that the pseudo-metric ρ n is weaker than the L ∞ norm, which means that our result does not follow from his.
Instead, we need to use a more intricate construction for the set of convex polyhedrons, and rely on the Varshamov-Gilbert lemma (Lemma 8).
First, we show that if we sample n points ( X i ) ni =1 i.i.d.
from the uniform distribution over the unit ball of R d , with constant probability (around 12 ) there exists a δ -packing of the unit ball of R d with k ≈ δ − d/ 2points.
For any n ∈ Z + , we define the set-valued random variable
[Equation]
, and the random variable A n = | S n | .
That is, A n is the number of points
X i that are at least epsilon away of any point with a lower index; clearly the set of such points constitutes a δ -packing of the unit ball of R d .
We have that E
[ A n | ( X i ) n − 1 i =1 ] =
A n − 1 + Pr ( ∀ 1 ≤ i ≤ n − 1 , ∥ X n − X i ∥
2 ≥ δ ) .
Since for a fixed X i and uniformly distributed X n , Pr ( ∥ X n − X i ∥ 2 ≤ δ ) ≤ δ d , a union bound shows that
Pr ( ∀ 1 ≤ i ≤ n − 1 , ∥ X n − X i ∥ 2 ≥ δ ) ≥ 1 − ( k − 1) δ d .
Thus, by the tower property of conditional expectations:
[Equation]
A simple induction shows that E [ A n ] ≥ n − (cid:80) n − 1 i =
iδ d = n − n ( n
[Equation]
This is a quadratic function of n .
For a given δ > 0 , we choose n that maximizes this expression, which results in
[Equation]
where we used ∼ instead of = to remark that n must be an integer.
Since A n is lower-bounded byand upper-bounded by n ∼ δ − d + 12 , Markov’s inequality shows that
[Equation]
where the approximation works for δ ≪ 1 .
Taking k
=
A n , this shows the existence of a δ -packing of the unit ball of size ≈ δ − d/ 2with probability at least (around) 12 .
Next, we use a construction similar to the proof of the lower bound in Thm.
6 of Bronshtein (1976).
That is, we consider the set ˜ S n = { ( x, g ( x ))
| x ∈ S n } ⊆
R d
+1 , where g : [ − 1 , 1] d → R is the map that parameterizes part of the surface of the ( d − 1) -dimensional
hypersphere S d − 1 ( R, t ) centered at (0 , . .
. , 0 , t ) of radius R for well chosen t, R .
As in the proof of Thm.
6 of Bronshtein (1976),
if x ∈ ˜ S n ⊆ S d − 1 ( R, t ) and we let S d − 1 ( R, t ) x denote the tangent space at x , we construct a hyperplane P that is parallel to S d − 1 ( R, t ) x at a distance ϵ < R from the latter and that intersects the sphere.
A simple trigonometry exercise involving the lengths of the chord and the sagitta of a two-dimensional circular segment shows that any point in
S d − 1 ( R, t ) that is at least √
Rϵ away from x will be separated from x by the hyperplane P .
Thus, the convex hull of ˜ S n \ { x } is at least ϵ away from x .
For any subset S ′ ⊆
˜ S n , we define the function g S ′
as the function whose epigraph (the set of points on or above the graph of a convex function) is equal to the convex hull of S ′ .
Note that g S ′ is convex and piecewise affine.
Let us set δ = √
Rϵ .
If x is a point in ˜ S n \ S ′ , by the argument in the previous paragraph the convex hull of S ′ is at least ϵ away from x .
Thus, | g S ′
( x ) − g S ′ ∪{ x } | ≥ ϵ .
The functions g S ′ and g S ′′ will differ by at least ϵ at each point in the symmetric difference S ′ ∆ S ′′ .
By the Varshamov-Gilbert lemma (Lemma 8), there is a set of 2 k/ 8 different subsets S ′
such that g S ′ differ pairwise at at least k/ 4 points in ˜ S n by at least ϵ .
Thus, for any S ′ ̸ = S ′′ in this set, we have that
[Equation]
We have to make sure that all the functions g S ′ belong to the set F d ( M, C ) .
Since | ddx √ R 2 − x 2 | = | x | √ R 2 − x 2 , the function g will be C -Lipschitz on [ − 1 , 1]
d if we take R 2 ≥ d (1+ C ) C , and in that case g S ′ will be Lipschitz as well for any S ′ ⊆
˜ S n .
To make sure that the uniform bound ∥ g S ′ ∥ ∞ ≤ M holds, we adjust the parameter t .
To obtain the statement of the proposition we start from a certain n and set δ such that (18) holds: δ = n − 1 d , and we set k ≈ n 2 .
Since δ = √
Rϵ , we have that ϵ
= 12 R δ 2 ≈ 12 (cid:113)
C d [Equation] 1+ C ) n − 2 d , which means that
[Equation]
Lemma 8 (Varshamov-Gilbert) .
Let N ≥ 8 .
There exists a subset Ω ⊆ { 0 , 1 } N such that | Ω | ≥ 2 N/and for any x, x ′ ∈
Ω such that x ̸ = x ′ , at least N/ 4 of the components differ.
Lemma 9.
For any ϵ > 0 , the packing number of the centered function class F d ( M, C ) =
{ f − E µ [ f ] | f ∈ F d ( M, C ) } with respect to the metric ρ n fulfills
[Equation]
Proof.
Let S be an ϵ -packing of F d ( M, C ) with respect to ρ n (i.e. |S| =
M ( ϵ ; F d ( M, C ) , ρ n ) ).
Let
F d ( M, C, t, δ ) = { f ∈ F d ( M, C ) | | E µ [ f ] − t | ≤ δ } .
If we let ( t i ) mi
=1 be a δ packing of [ − C, C ] , we can write F d ( M, C ) =
∪ m i =1 F d ( M, C, t i , δ ) .
By the pigeonhole principle, there exists an index i ∈ { 1 , . . .
, m } such that |S ∩ F d ( M, C, t i , δ ) | ≥ | S | /m .
Let F d ( M, C, t i )
= { f ∈ F d ( M, C ) | E µ [ f ] = t i } , and let P : F d ( M, C )
→ F d ( M, C, t i ) be the projection operator defined as f (cid:55)→ f − E µ [ f ] + t i .
If we take δ = ϵ/ (4 n ) , we have that ρ n ( Pf, Pf ′ ) ≥ ϵ/ 2 for any f ̸ = f ′ ∈ S ∩ F d ( M, C, t i , δ ) , as ρ n ( Pf, Pf ′ ) ≤ ρ n ( Pf, f ) + ρ n ( f, f ′ )
+ ρ n ( f ′ , Pf ′ ) , and ρ n ( f, Pf ) ≤ n | E µ [ f ] − t i | ≤ nδ = ϵ/ 4 , while ρ n ( f, f ′ ) ≥ ϵ .
Thus, the ϵ/ 2 -packing number of F d ( M, C, t i ) is lower-bounded by |S ∩ F d ( M, C, t i , δ ) | ≥ | S | /m .
Since m =
⌊ (2 C + 2 δ ) / (2 δ ) ⌋ ≤ 4 nC/ϵ + 1 , this is lower-bounded by | S | / (4 nC/ϵ + 1) .
The proof concludes using the observation that the map F d ( M, C, t i ) → ¯ F d ( M, C ) defined as f (cid:55)→ f − t i is an isometric bijection with respect to ρ n .
Corollary 1.
With probability at least 1 / 2 , the packing number of the set F d ( M, C ) with respect to the metric ρ n satisfies
[Equation]
[Table]
Proof of Theorem 5.
Lemma 5 provides upper and lower bounds to E
[ d CT , A ( µ, µ n )] in terms of the Rademacher complexities of A and its centered version, ¯ A .
Lemma 1 in App.
A states that A is equal to the space F d ( M, C ) of convex functions on [ − 1 , 1] d such that | f ( x ) | ≤ M and | f
( x ) − f ( y ) | ≤ C ∥ x − y ∥ for any x, y ∈ [ − 1 , 1] d .
Let E X,ϵ
[ ∥ S n ∥ F ] , E X,ϵ
[ ∥ G n ∥ F ] be the
Rademacher and Gaussian complexities of a function class F (see definitions in Lemma 5 and
Lemma 6).
We can write
[Equation]
where the first inequality holds by Lemma 6, and the second inequality follows from Proposition 8.
This gives rise to (7) upon redefining C 0 ← C 0 / √ 8 .
Equation (8) follows from the Rademacher complexity upper bound in Lemma 5, and the upper bound on the empirical Rademacher complexity of F d ( M, C ) given by Proposition 4.
Proof of Theorem 6.
We upper-bound E [ d CT ,F L, M ,k, + (1) ( µ, µ n )] as in the upper bound of E [ d CT , A ( µ, µ n )] in Lemma 5:
E [ d CT ,F L, M ,k, + (1) ( µ, µ n )] ≤ 2 E X
[Equation]
Likewise, we get that E [VDC F L, M ,k, + (1) ( µ, µ n )] ≤ 2 E ϵ
[ ∥ S n ∥ F L, M ,k, + (1) ] .
Since F L, M ,k, + (1) ⊂
F L, M ,k (1) , we have that E ϵ
[ ∥ S n ∥ F L, M ,k, + (1) ] ≤ E ϵ [ ∥ S n ∥ F L, M ,k (1) ] .
Then, the result follows from the Rademacher complexity upper bound from Proposition 7.
E S IMPLE EXAMPLES IN DIMENSION 1 FOR VDC AND d CT
For simple distributions over compact sets of R , we can compute the CT discrepancy exactly.
Let η : R → R be a non-negative bump-like function, supported on [ − 1 , 1] , symmetric w.r.t 0, increasing on ( − 1 , 0] , and such that (cid:82) R η = 1 .
Clearly η is the density of some probability measure with respect to the Lebesgue measure.
Same variance, different mean.
Let µ + ∈ P
( R )
with density η ( x − a ) for some a > 0 , and let µ − ∈ P ( R ) with density η ( x + a ) .
We let K =
[ − C, C ] for any C > 0 .
Proposition 10.
Let F ( x ) =
(cid:82) x
−∞ ( η ( y + a ) − η ( y − a )) dy , and G ( x ) = (cid:82) x 0 F ( y ) dy .
We obtain that
VDC K ( µ + || µ − )
= 2 CG (+ ∞ ) .
The optimum of ( P 2 ) is equal to the Dirac delta at − C : ν = δ − C , while an optimum of ( P 1 ) is u = − Cx .
We have that VDC K ( µ + || µ − )
= 2 CG (+ ∞ ) as well, and thus, d CT , A ( µ − , µ + ) = 4 CG (+ ∞ ) .
Proof.
Suppose that first that a > 0 .
We have that F ( −∞ ) =
F ( − a − 1) = 0 , F (+ ∞ ) =
F ( a + 1) = 0 .
F is even, and sup x | F ( x ) | = 1 .
We have that G is odd (in particular G ( − a − 1) =
− G ( a + 1) ) and non-decreasing on R .
Also, 0 < G (+ ∞ ) =
G ( a + 1) ≤ a + 1 .
Let K =
[ − C, C ] where C > 0 , and let Ω =
[ − a − 1 , a + 1] .
For any twice-differentiable 2 convex function u on Ω such that u ′ ∈ K , we have
[Equation]
If we reexpress u ( a + 1)
=
u ′ ( − a − 1) + (cid:82) a +1 − a − 1 u ′′ ( x ) dx , the right-hand side becomes
[Equation]
Since G is increasing, for any x ∈ [ − a − 1 , a + 1) , G ( x ) < G ( a + 1) .
The convexity assumption on u implies that u ′′ ≥ 0 on Ω , and the condition u ′ ∈ K means that u ′ ∈ [ − C, C ] .
Thus, the function u that maximizes (19) fulfills u ′ ≡ − C, u ′′ ≡ 0 .
Thus, we can take u = − Cx .
The measure ν = ( ∇ u ) # µ − =
( ∇ u ) # µ + is equal to the Dirac delta at − C : ν = δ − C .
Hence, sup u ∈A (cid:82) Ω u d ( µ − − µ + ) = 2 CG ( a + 1) .
Thus, VDC K ( µ + || µ − )
=
2 CG ( a + 1) .
Reproducing the same argument yields VDC K ( µ − || µ + )
= 2 CG ( a + 1) , and hence the CT distance is equal to d CT , A ( µ − , µ + )
= 4 CG ( a + 1) .
Same mean, different variance.
Let µ + ∈ P ( R ) with density 1 a η ( xa ) for some a > 0 , and let µ − ∈ P
( R )
with density η ( x ) .
Note that µ + has support [ − a, a ] , and standard deviation equal to a times the standard deviation of µ − .
We let K =
[ − C, C ] for any C > 0 as before.
Proposition 11.
Let F ( x ) =
(cid:82) x
−∞ ( η ( y ) − 1 a η ( ya )) dy , and G ( x ) = (cid:82) x
−∞ F ( y ) dy .
When a < 1 , we have that VDC K ( µ + || µ − )
= 2 CG (0) .
The optimum of ( P 2 ) is equal to ν = 12 δ − C + 12 δ C , while an optimum of ( P 1 ) is u = C | x | .
When a > 1 , VDC K ( µ + || µ − ) = 0 .
For any a > 0 , d CT , A ( µ − , µ + ) = 2 CG (0) .
Proof.
We define Ω =
[ − max { a, 1 } , max { a, 1 } ] and K = [ − C, C ] .
We have that F ( −∞ ) =
F ( − max { a, 1 } ) = 0 , F (+ ∞ ) =
F (max { a, 1 } ) = 0 , and F (0)
= 0 .
F is odd.
When a <it is non-positive on [0 , + ∞ ) and non-negative on ( −∞ , 0] . When a > 1 , it is non-negative on [0 , + ∞ ) and non-positive on ( −∞ , 0] .
We have that G ( −∞ ) =
G ( − max { a, 1 } ) =
and
G (+ ∞ ) =
G (max { a, 1 } ) = 0 .
G is even.
When a < 1 , G is non-negative, non-decreasing on ( −∞ , 0] and non-increasing on [0 , + ∞ ) : it has a global maximum at 0.
When a > 1 , G is nonpositive, non-increasing on ( −∞ , 0] and non-decreasing on [0 , + ∞ ) : it has a global minimum at 0.
We have that
[Equation] [Equation]
Thus, when a < 1 this expression is maximized when u ′′ ∝ δ 0 .
Taking into account the constraints u ′ ∈ [ − C, C ] , the optimal u ′ is u ′ ( x ) =
C sign ( x ) , which means that an optimal u is u ( x ) =
C | x | .
Thus, the measure ν = ( ∇ u ) # µ − = ( ∇ u ) # µ + is equal to the average of Dirac deltas at − C and C : ν = 12 δ − C + 12 δ C .
We obtain that
[Equation]
When a > 1 , the expression (20) is maximized when u ′′ = 0 , which means that any u ′ constant and any u affine work.
Any measure ν concentrated at a point in [ − C, C ] is optimal.
Thus, sup u ∈A (cid:82) Ω u d ( µ − − µ + ) = 0 .
We conclude that d CT , A ( µ + , µ − )
= 2 CG (0) .
F A LGORITHMS FOR LEARNING DISTRIBUTIONS WITH SURROGATE VDC AND CT DISTANCE
In Algorithms 1 and 2, we present the steps for learning with the VDC and the CT distance, respectively.
In order to enforce convexity of the ICMNs, we leverage the projected gradient descent algorithm after updating hidden neural network parameters.
Additionally, to regularize the u networks in Algorithm 1, we include a term that penalizes the square of the outputs of the Choquet critic on the baseline and generated samples (Mroueh & Sercu, 2017).
Input: Target distribution ν , baseline g 0 , latent distribution µ 0 , integer maxEpochs , integer discriminatorEpochs , Choquet weight λ , GAN learning rate η , Choquet critic learning rate η VDC ,
Choquet critic regularization weight λ u,reg , WGAN gradient penalty regularization weight λ GP
Initialize: untrained discriminator f φ , untrained generator g θ , untrained Choquet ICMN critic u ψ ψ ← ProjectHiddenWeightsToNonNegative() for i = 1 to maxEpochs do
L
[Equation]
L
[Equation]
Y ∼ µ 0 ( ||∇ x f φ ( tg θ ( Y ) + (1 − t ) X )
|| − 1) 2 ]
L VDC ( ψ, θ ) =
E Y ∼ µ
[ u ψ ( g θ ( Y ))
− u ψ ( g 0 ( Y ))] for j = 1 to discriminatorEpochs do φ ← φ − η ∇ φ ( L WGAN + λ GP L GP ) { ADAM optimizer }
L VDC reg =
E Y ∼ µ
[ u ψ ( g θ ( Y )) 2 + u ψ ( g 0 ( Y )) 2 ] ψ ← ψ − η VDC ∇ ψ
( L VDC + λ u,reg L VDC reg ) { ADAM optimizer } ψ ← ProjectHiddenWeightsToNonNegative() end for θ ← θ + η ∇ θ ( L WGAN + λL VDC ) { ADAM optimizer } end for
Return g θ
G P ROBING MODE COLLAPSE
As described in Sec. 7, to investigate how training with the surrogate VDC regularizer helps alleviate mode collapse in GAN training, we implemented GANs trained with the IPM objective alone and compared this to training with the surrogate VDC regularizer for a mixture of 8 Gaussians target distribution.
To track mode collapse, we report two metrics: 1)
The mode collapse score that we define as follows: for each generated point, we assign it to the nearest neighbor cluster in the target mixture and obtain a histogram over the modes computed on all generated points.
The closer this histogram is to the uniform distribution, the less mode collapsed is the generator.
We quantify this with the KL distance of this histogram to the uniform distribution on 8 modes.
The negative log likelihood (NLL) of the Gaussian mixture.
A converged generator needs to have a low negative likelihood and low mode collapse score.
In Figure 3, we see that in the baseline training (unregularized GAN training), we observe mode collapse and cycling between modes, evidenced by the fluctuating mode collapse score and NLL.
In contrast, when training with the VDC regularizer to improve upon the collapse generator g 0 (which is taken from step 55k from the unregularized GAN training), we see more stable training and better mode coverage.
As the regularization weight λ VDC for VDC increases, the dominance constraint is more strongly enforced resulting in a better NLL and smaller mode collapse score.
Input: Target distribution ν , latent distribution µ 0 , integer maxEpochs , integer criticEpochs , learning rate η
Initialize: untrained generator g θ , untrained Choquet ICMN critics u ψ 1 , u ψ 2 ψ 1 ← ProjectHiddenWeightsToNonNegative() ψ 2 ← ProjectHiddenWeightsToNonNegative() for i = 1 to maxEpochs do
L D CT , 1 ( ψ 1 , θ ) =
E Y ∼ µ
[ u ψ 1 ( g θ ( Y ))] − E X ∼ ν
[ u ψ 1 ( X )]
L
D CT , 2 ( ψ 2 , θ ) =
E X ∼
ν
[ u ψ 2 ( X )] − E Y ∼ µ
[ u ψ 2 ( g θ ( Y ))]
L d CT ( ψ 1 , ψ 2 , θ ) =
L D CT , 1 ( ψ 1 , θ ) + L D CT , 2 ( ψ 2 , θ ) for j = 1 to discriminatorEpochs do ψ 1 ← ψ 1 − η ∇ ψ 1 L D CT , 1 { ADAM optimizer } ψ 1 ← ProjectHiddenWeightsToNonNegative() ψ 2 ← ψ 2 − η ∇ ψ 2 L D CT , 2 { ADAM optimizer } ψ 2 ← ProjectHiddenWeightsToNonNegative() end for θ ← θ + η ∇ θ L d CT { ADAM optimizer } end for
Return g θ
H A DDITIONAL EXPERIMENTAL DETAILS
Portfolio optimization In Figure 4 we plot the trajectory of the z parameter from the portfolio optimization example solved in Sec. 7.
The convex decreasing u network was parameterized with a 3-layer, fully-connected, decreasing
ICMN with hidden dimension 32 and maxout kernel size of 4.
See Table 2 for full architectural details.
Optimization was performed on CPU.
[Table]
Image generation In training WGAN-GP and WGAN-GP + VDC, we use residual convolutional neural network (CNN) for the generator and CNN for the discriminator (both with ReLU nonlinearities).
See Table 3, Table 4, and Table 5 for the full architectural details.
Note that in Table 3,
PixelShuffle refers to a dimension rearrangement where an input of dimension Cr 2 × H × W is rearranged to C ×
Hr × Wr 3 .
Our latent dimension for µ 0 is 128 and λ GP = 10 .
We use ADAM optimizers (Kingma & Ba, 2015) for both networks, learning rates of 1e − 4 , and a batch size of 64.
We use the CIFAR-10 training data and split it as 95% training and 5% validation.
FID is calculated using the validation set.
The generator was trained every 6th epoch, and training was executed for about 400 epochs in total.
When training g ∗ we use learning rate of 1e − 5 for the Choquet critic, λ = 10 , and λ u,reg = 10 .
Training the baseline g 0 and g ∗ with the surrogate VDC was done on a compute environment with 1 CPU and 1 A100 GPU.
2D point cloud generation When training with the d CT surrogate for point cloud generation, the generator and Choquet critics are parameterized by residual maxout networks with maxout kernel size of 2.
The critics are ICMNs.
Our latent dimension for µ 0 is 32.
Both the generator and critics have hidden dimension of 32.
The generator consists of 10 fully-connected layers and the critics consists of 5.
For all networks, we add residual connections from input-to-hidden layers (as opposed to hidden-to-hidden).
The last layer for all networks is fully-connected linear.
See Table 6 for full architectural details.
We use ADAM optimizers for all networks, learning rate of 5e − 4 for the generator, learning rates of 1e − 4 for the Choquet critics, and a batch size of 512.
Training was done on a single-CPU environment.
[Table] [Table]
[Table] [Table]
I A SSETS
Libraries Our experiments rely on various open-source libraries, including pytorch (Paszke et al., 2019)
(license: BSD) and pytorch-lightning (Falcon et al., 2019) (Apache 2.0).
Code re-use For several of our generator, discriminator, and Choquet critics, we draw inspiration and leverage code from the following public Github repositories: (1) https://github.com/caogang/wgan-gp , (2) https://github.com/ozanciga/ gans-with-pytorch , and (3) https://github.com/CW-Huang/CP-Flow .
Data In our experiments, we use following publicly available data: (1) the CIFAR-10 (Krizhevsky &
Hinton, 2009) dataset, released under the MIT license, and (2) the Github icon silhouette, which was copied from https://github.com/CW-Huang/CP-Flow/blob/main/imgs/github.
png .
CIFAR-10 is not known to contain personally identifiable information or offensive content.