A M EAN -F IELD T HEORY FOR K ERNEL A LIGNMENT WITH R ANDOM F EATURES IN G ENERATIVE A DVERSERIAL N ETWORKS
Anonymous authors
Paper under double-blind review
A BSTRACT
We propose a novel supervised learning method to optimize the kernel in maximum mean discrepancy generative adversarial networks (MMD GANs).
Specifically, we characterize a distributionally robust optimization problem to compute a good distribution for the random feature model of Rahimi and Recht to approximate a good kernel function.
Due to the fact that the distributional optimization is inﬁnite dimensional, we consider a Monte-Carlo sample average approximation (SAA) to obtain a more tractable ﬁnite dimensional optimization problem.
We subsequently leverage a particle stochastic gradient descent (SGD) method to solve the derived ﬁnite dimensional optimization problem.
Based on a mean-ﬁeld analysis, we then prove that the empirical distribution of the interactive particles system at each iteration of the SGD follows the path of the gradient descent ﬂow on the Wasserstein manifold.
We also establish the non-asymptotic consistency of the ﬁnite sample estimator.
Our empirical evaluation on synthetic data-set as well as MNIST and CIFAR-10 benchmark data-sets indicates that our proposed
MMD GAN model with kernel learning indeed attains higher inception scores well as Fr ` echet inception distances and generates better images compared to the generative moment matching network (GMMN) and MMD GAN with untrained kernels.
I NTRODUCTION
A fundamental and long-standing problem in unsupervised learning systems is to capture the underlying distribution of data.
While deep generative models such as Boltzmann machines Salakhutdinov & Hinton (2009) and auto-encoding variational Bayes Kingma & Welling (2013) accomplish this task to some extent, they are inadequate for many intractable probabilistic computations that arise in maximum likelihood estimation.
Moreover, in many machine learning tasks such as caption generation Xu et al. (2015), the main objective is to obtain new samples rather than to accurately estimate the underlying data distribution.
Generative adverserial network (GAN) Goodfellow et al.
(2014) provides a framework to directly draw new samples without estimating data distribution.
It consists of a deep feedforward network to generate new samples from a base distribution ( e.g. Gaussian distribution), and a discriminator network to accept or reject the generated samples.
However, training GAN requires ﬁnding a Nash equilibrium of a non-convex minimax game with continuous, high-dimensional parameters.
Consequently, it is highly unstable and prone to miss modes Salimans et al.
; Che et al. (2016).
To obtain more stable models, the generative moment matching networks (GMMNs)
Li et al.
(2015) are proposed, wherein instead of training a discriminator network, a non-parametric statistical hypothesis test is performed to accept or reject the generated samples via the computation of the kernel maximum mean discrepancy Gretton et al.
While leveraging a statistical test simpliﬁes the loss function for training GMMN, in practice, the diversity of generated samples by GMMN is highly sensitive to the choice of the kernel.
Thus, to improve the sampling performance, the kernel function also needs to be jointly optimized with the generator.
Rather than i optimizing the kernel directly, the MMD GAN model Li et al.
(2017) is proposed in which an embedding function is optimized in conjunction with a ﬁxed user-deﬁned kernel ( e.g. RBF Gaussian kernel).
However, there are no theoretical guarantees that the user-deﬁned kernel is the ‘right’ kernel for embedded features.
Contributions .
To address the kernel model selection problem in MMD GAN Li et al.
(2017), in this paper we put forth a novel framework to learn a good kernel function from training data.
Our kernel learning approach is based on a distributional optimization problem to learn a good distribution for the random feature model of Rahimi and Recht Rahimi & Recht (2008; 2009) to approximate the kernel.
Since optimization with respect to the distribution of random features is inﬁnite dimensional, we consider a Monte Carlo approximation to obtain a more tractable ﬁnite dimensional optimization problem with respect to the samples of the distribution.
We then use a particle stochastic gradient descent (SGD) to solve the approximated ﬁnite dimensional optimization problem.
We provide a theoretical guarantee for the consistency of the ﬁnite sample-average approximations.
Based on a mean-ﬁeld analysis, we also show the consistency of the proposed particle SGD.
In particular, we show that when the number of particles tends to inﬁnity, the empirical distribution of the particles in
SGD follows the path of the gradient descent ﬂow of the distributional optimization problem on the
Wasserstein manifold.
P RELIMINARIES OF MMD GAN S
Assume we are given data { v i } ni =1 that are sampled from an unknown distribution P V with the support V .
In many unsupervised tasks, we wish to attain new samples from the distribution P V without directly estimating it.
Generative Adversarial Network (GAN) Goodfellow et al.
(2014) provides such a framework.
In vanilla GAN, a deep network G ( · ; W ) parameterized by W ∈ W is trained as a generator to transform the samples Z ∼ P Z , Z ∈ Z from a user-deﬁned distribution
P Z ( e.g. Gaussian) into a new sample G ( Z ; W ) ∼ P W , such that the distributions P W and P V are close.
In addition, a discriminator network D ( · ; δ ) parameterized by δ ∈ ∆ is also trained to reject or accept the generated samples as a realization of the data distribution.
The training of the generator and discriminator networks is then accomplished via solving a minimax optimization problem as below
[Equation]
In the high dimensional settings, the generator trained via the min-max program of equation 1 can potentially collapse to a single mode of distribution where it always emits the same point Che et al.
To overcome this shortcoming, other adversarial generative models are proposed in the literature, which propose to modify or replace the discriminator network by a statistical two-sample test based on the notion of the maximum mean discrepancy which is deﬁned below:
Deﬁnition 2.1.
(M AXIMUM M EAN D ISCREPANCY G RETTON ET AL .
Let ( X , d ) be a metric space, F be a class of functions f : X → IR , and P,
Q ∈ B ( X ) be two probability measures from the set of all Borel probability measures B ( X ) on X .
The maximum mean discrepancy (MMD) between the distributions P and Q with respect to the function class F is deﬁned below
[Equation]
Different choices of the function class F in equation 2 yield different adversarial models such as Wasserstein GANs (WGAN)
Arjovsky et al.
(2017), f -GANs Nowozin et al.
(2016), and GMMN and MMD GAN Li et al.
In the latter two cases, the function class F def = { f : X →
IR : (cid:107) f
(cid:107) H X ≤ 1 } , where H X is a reproducing kernel Hilbert space (RKHS) of functions with a kernel K : X × X
→ IR , denoted by ( H X , K ) .
Then, the squared MMD loss in equation 2 as a ii measure of the distance between the distributions P V and P W has the following expression
[Equation]
where X = V ∪ W .
Instead of training the generator via solving the minimax optimization in equation 1, the MMD GAN model of Li et al.
(2015) proposes to optimize the discrepancy between two distributions via optimization of an embedding function ι : IR D (cid:55)→ IR p , p ≤ D , i.e. ,
[Equation]
where k : IR p × IR p → IR is a user-deﬁned ﬁxed kernel.
In Li et al.
, the proposal for the kernel k : IR p × IR p → IR is a mixture of the Gaussians,
[Equation]
where the bandwidth parameters σ 1 , · · · , σ m > 0 are manually selected.
Nevertheless, in practice there is no guarantee that the user-deﬁned kernel k ( ι ( x ) , q ( y )) can capture the structure of the embedded features ι ( x ) .
P ROPOSED APPROACH : KERNEL LEARNING WITH RANDOM FEATURES FOR MMD GAN S
In this section, we ﬁrst expound our kernel learning approach.
Then, we describe a novel MMD
GAN model based on the proposed kernel learning approach.
3. R OBUST DISTRIBUTIONAL OPTIMIZATION FOR KERNEL LEARNING
To address the kernel model selection issue in MMD GAN Li et al.
(2017), we consider a kernel optimization scheme with random features Rahimi & Recht (2008; 2009).
Let ϕ : IR d × IR D
→ [ − 1 , 1] denotes the explicit feature maps and µ ∈ M (IR D ) denotes a probability measure from the space of probability measures M (IR D ) on IR D .
The kernel function is characterized via the explicit feature maps using the following integral equation
[Equation]
[Equation]
Let MMD µ [ P V , P W ] def = MMD K µ [ P V , P W ] .
Then, the kernel optimization problem in can be formulated as a distribution optimization for random features, i.e ,
[Equation]
Here, P is the set of probability distributions corresponding to a kernel class K .
In the sequel, we consider P to be the distribution ball of radius R as below
[Equation]
[Equation]
where µ 0 is a user-deﬁned base distribution, and To establish the proof, we consider the W p ( · , · ) is the p -Wasserstein distance deﬁned as below
[Equation]
iii where the inﬁmum is taken with respect to all couplings π of the measures µ, µ 0 ∈ M (IR D ) , and Π( µ, µ 0 ) is the set of all such couplings with the marginals µ and µ 0 .
The kernel MMD loss function in equation 8 is deﬁned with respect to the unknown distributions of the data-set P V and the model P W .
Therefore, we construct an unbiased estimator for the MMD loss function in equation 8 based on the training samples.
To describe the estimator, sample the labels from a uniform distribution y 1 , · · · , y n ∼ i.i.d.
Uniform {− 1 , +1 } , where we assume that the number of positive and negative labels are balanced.
In particular, consider the set of positive labels
I = { i ∈ { 1 , 2 , · · · , n } : y i = +1 } , and negative labels J = { 1 , 2 , · · · , n } / I , where their cardinality is |I| =
|J | = n 2 .
We consider the following assignment of labels:
• Positive class labels :
If y i =
+1 , sample the corresponding feature map from data-distribution x i
= v
i ∼ P V .
• Negative class labels :
If y i = − 1 , sample from the corresponding feature map from the generated distribution x i =
G ( Z i , W ) ∼ P W ,
Z i ∼ P Z .
By this construction, the joint distribution of features and labels P Y, X has the marginals P X | Y
=+1 =
P
V , and P X | Y = − 1 =
P W .
Moreover, the following statistic, known as the kernel alignment in the literature (see, e.g. , Sinha & Duchi (2016); Cortes et al.
(2012)), is an unbiased estimator of the
MMD loss in equation 8,
[Equation]
[Equation]
See Appendix C.1 for the related proof.
The kernel alignment in equation 11 can also be viewed through the lens of the risk minimization
[Equation]
Here, α > 0 is a scaling factor that determines the separation between feature vectors, and
K ∗ def = α yy T is the ideal kernel that provides the maximal separation between the feature vectors over the training data-set, i.e. , K ∗ ( x i , x j ) =
α when features have identical labels y i =
y j , and K ∗ ( x i , x j )
= − α otherwise.
Upon expansion of the risk function in equation 12, it can be easily shown that it reduces to the kernel alignment in equation 11 when α → + ∞ .
Intuitively, the risk minimization in equation 12 gives a feature space in which pairwise distances are similar to those in the output space Y =
{− 1 , +1 } .
3. SAA FOR DISTRIBUTIONAL OPTIMIZATION
The distributional optimization problem in equation 8 is inﬁnite dimensional, and thus cannot be solved directly.
To obtain a tractable optimization problem, instead of optimizing with respect to the distribution µ of random features, we optimize the i.i.d.
samples (particles) ξ 1 , · · · , ξ N ∼ i.i.d.
µ generated from the distribution.
The empirical distribution of these particles is accordingly deﬁned as follows
[Equation] [Equation]
where δ ( · ) is the Dirac’s delta function concentrated at zero.
In practice, the optimization problem in equation 12 is solved via the Monte-Carlo sample average approximation of the objective function,
[Equation]
iv where P N def = IB NR
[Equation]
µ N 0 )
= (cid:110)(cid:98) µ N ∈ M
(IR D ) : W p ( (cid:98) µ N , (cid:98) µ N 0 ) ≤ R (cid:111) , and (cid:98) µ N 0 is the empirical measure associated with the initial samples ξ 10 , · · · , ξ N 0 ∼ i.i.d.
µ 0 .
The empirical objective function in equation 14 can be optimized with respect to the samples ξ 1 , · · · , ξ N using the particle stochastic gradient descent.
For the optimization problem in equation 14, the (projected) stochastic gradient descent (SGD) takes the following recursive form,
[Equation]
for k = 1 , 2 , · · · , N , where ( y m , x m ) , ( (cid:101) y m , (cid:101) x m ) ∼ i.i.d P x ,y and η ∈ IR > 0 denotes the learning rate of the algorithm, and the initial particles are ξ 10 , · · · , ξ N 0 ∼ i.i.d.
µ 0 .
At each iteration of the SGD dynamic in equation 15, a feasible solution for the inner optimization of the empirical risk function in equation 14 is generated via the empirical measure
[Equation]
[Equation]
Indeed, we prove in Section 4 that for an appropriate choice of the learning rate η > 0 , the empirical measure in equation 16 remains inside the distribution ball (cid:98) µ Nm ∈ P N for all m ∈ [0 , NT ] ∩ IN , and is thus a feasible solution for the empirical risk minimization equation 14 (see Corollary 4.2.
in Section 4).
3. P ROPOSED MMD GAN WITH KERNEL LEARNING
In Algorithm 1, we describe the proposed method MMD GAN model with the kernel learning approach described earlier.
Algorithm 1 has an inner loop for the kernel training and an outer loop for training the generator, where we employ RMSprop Tieleman & Hinton (2012).
Our proposed
MMD GAN model is distinguished from MMD GAN of Li et al.
(2017) in that we learn a good kernel function in equation 17 of the inner loop instead of optimizing the embedding function that is implemented by an auto-encoder.
However, we mention that our kernel learning approach is compatible with the auto-encoder implementation of Li et al.
(2017) for the dimensionality reduction of features (and particles).
In the case of including an auto-encoder, the inner loop in Algorithm 1 must be modiﬁed to add an additional step for training the auto-encoder.
However, to convey the main ideas more clearly, the training step of the auto-encoder is omitted from Algorithm 1.
3. C OMPUTATIONAL C OMPLEXITY A NALYSIS
Sampling the labels y, (cid:101) y ∼ i.i.d.
Uniform {− 1 , +1 } require O (1) complexity, while sampling (cid:101) x
| (cid:101) y
= +1 , x | y
= +1 ∼ P V and (cid:101) x
| (cid:101) y
= +1 , x | y
=
− 1 ∼ P W has a complexity of O ( d ) .
The computation of the stochastic gradient step in equation 17 requires computing the sum 1 N (cid:80) Ni =1 ϕ ( x ; ξ k ) ϕ
( (cid:101) x ; ξ k ) .
This can be done as a separate preprocessing step prior to executing the SGD in equation 17, and requires preparing the vectors ϕ def =
[ ϕ ( x ; ξ 1 ) , · · · , ϕ ( x ; ξ N )] and (cid:101) ϕ
def =
[ ϕ ( (cid:101) x ; ξ 1 ) , · · · , ϕ ( (cid:101) x ; ξ N )] .
Using the random feature model of Rahimi and Recht Rahimi &
Recht (2008; 2009), where ϕ ( x ; ξ ) = √ 2 cos( x T ξ + b ) .
Here b ∼ Uniform[ − π, + π ] , the complexity of computing the vectors ϕ and (cid:101) ϕ
is of the order O ( Nd ) on a single processor.
However, this construction is trivially parallelizable.
Furthermore, computation can be sped up even further for certain distributions µ 0 .
For example, the Fastfood technique can approximate ϕ and ˜ ϕ in O ( N log( d )) v
[Table]
while time for the Gaussian kernel Le et al.
Updating each particle in equation 17, involves the computation of the gradient ∇ ξ ( ϕ ( ξ ; ξ k ) ϕ
( (cid:101) ξ ; ξ k ))) which is O ( d ) .
Thus, the complexity of one iteration of SGD for all the particles is O ( Nd ) .
Overall, one step of the kernel learning has a complexity of O ( Nd ) .
On the other hand, to attain ε -suboptimal solution max k =1 , 2 , ··· ,N (cid:107) ξ k − ξ k ∗ (cid:107) ≤ ε , the SGD requires has the sample complexity
O (cid:0) log( 1 ε ) (cid:1) .
Consequently, the computational complexity of the kernel learning is of the order of
O ( Nd log( 1 ε )) .
To compare this complexity with that of MMD GAN is of the order O (cid:0) B 2 (cid:96) log( 1 ε ) (cid:1) , where B is the batch size for approximation of the population MMD, and (cid:96) is the number of kernel mixtures.
C ONSISTENCY AND A MEAN - FIELD ANALYSIS
In this section, we provide theoretical guarantees for the consistency of various approximations we made to optimize the population MMD loss function in equation 8.
We defer the proofs of the following theoretical results to AppendixC. The main assumptions (( A.1 ),( A.2 ), and ( A.3 )) underlying our theoretical results are also stated in the same section.
Consistency of ﬁnite-sample estimate : In this part, we prove that the solution to ﬁnite sample optimization problem in equation 14 approaches its population optimum in equation 8 as the number of data points as well as the number of random feature samples tends to inﬁnity.
Theorem 4.1.
(N ON - ASYMPTOTIC C ONSISTENCY OF F INITE -S AMPLE E STIMATOR )
Suppose conditions ( A . 1 ) - ( A . 3 ) of Appendix C are satisﬁed.
Consider the distribution balls P and P N that are deﬁned with respect to the 2 -Wasserstein distance ( p = 2) .
Furthermore, consider the optimal MMD values of the population optimization and its ﬁnite sample estimate
[Equation]
vi respectively.
Then, with the probability of (at least) 1 − 3 (cid:37) over the training data samples { ( x i , y i ) } ni =1 and the random feature samples { ξ k 0 } Nk =1 , the following non-asymptotic bound holds
[Equation]
where c 1 = 3 14 × 2 4 , and c 2 = 9 × 2 11 .
The proof of Theorem 4.1 is presented in Appendix C.1.
Notice that there are three key parameters involved in the upper bound of Theorem 4.1.
Namely, the number of training samples n , the number of random feature samples N , and the regularization parameter α .
The upper bound in equation 20 thus shows that when n, N, α → + ∞ , the solution obtained from solving the empirical risk minimization in equation 12 yields a MMD population value tending to the optimal value of the distributional optimization in equation 8.
Consistency of particle SGD for solving distributional optimization .
The consistency result of Theorem 4.1 is concerned with the MMD value of the optimal empirical measure (cid:98) µ N ∗ ( ξ ) =
N (cid:80) Nk
=1 δ ( ξ − ξ k ∗ ) of the empirical risk minimization equation 14.
In practice, the particle SGD is executed for a few iterations and its values are used as an estimate for ( ξ 1 ∗ , · · · , ξ N ∗ ) .
Consequently, it is desirable to establish a similar consistency type result for the particle SGD estimates ( ξ 1 m , · · · , ξ Nm ) at the m -th iteration.
To reach this objective, we deﬁne the scaled empirical measure as follows
[Equation]
At any time t , the scaled empirical measure µ Nt is a random element, and thus ( µ Nt ) 0 ≤ t ≤ T is a measured-valued stochastic process.
Therefore, we characterize the evolution of its Lebesgue density p
Nt ( ξ ) def = µ Nt (d ξ ) / d ξ in the following theorem:
Theorem 4.2.
(M C K EAN -V LASOV M EAN -F IELD PDE) Suppose conditions ( A . 1 ) -
( A . 3 ) of Appendix C are satisﬁed.
Further, suppose that the Radon-Nikodyme derivative q 0 ( ξ ) = µ 0 (d ξ ) / d ξ exists.
Then, there exists a unique solution ( p ∗ t ( ξ )) 0 ≤ t ≤ T to the following non-linear partial differential equation
[Equation]
[Equation]
Moreover, the measure-valued process { ( µ Nt ) 0 ≤ t ≤
[Equation]
} N ∈ IN deﬁned in equation 21 converges (weakly) to the unique solution µ ∗ t ( ξ ) =
p ∗ t ( ξ )d ξ as the number of particles tend to inﬁnity N → ∞ ..
Due to the mean-ﬁeld analysis of Theorem 4.2, we can prove that the empirical measure (cid:98) µ Nm of the particles in SGD dynamic equation 15 remains inside the feasible distribution ball P N :
Corollary 4.2.1.
Consider the learning rate η
= O (cid:16)
[Equation]
(cid:17) for the SGD in equation 15.
Then, the empirical measure (cid:98) µ Nm of the particles remains inside the distributional ball (cid:98) µ Nm ∈ P N = { (cid:98) µ
N ∈ M
(IR D ) : W p ( (cid:98) µ N , (cid:98) µ N 0 ) ≤ R } for all m ∈ [0 , NT ] ∩ IN , with the probability of (at least) 1 − δ .
vii
Let us make two remarks about the PDE in equation 22.
First, the seminal works of Otto Otto (2001), and Jordan, et al .
Jordan et al.
(1998) establishes a deep connection between the McKeanVlasov type PDEs speciﬁed in equation 22 and the gradient ﬂow on the Wasserstein manifolds.
More speciﬁcally, the PDE equation in equation 22 can be thought of as the minimization of the energy functional
[Equation]
using the following gradient ﬂow dynamics
[Equation]
where grad p t E ( p t ( ξ ))
= ∇ ξ · ( p t ( ξ ) ∇ ξ R α
[Equation]
p t (
[Equation]
))) is the Riemannian gradient of R α ( µ t ( ξ )) with respect to the metric of the Wasserstein manifold .
This shows that when the number of particles in particle SGD equation 15 tends to inﬁnity ( N → + ∞ ) , their empirical distribution follows a gradient descent path for minimization of the population version (with respect to data samples) of the distributional risk optimization in equation 12.
In this sense, the particle SGD is the ‘consistent’ approximation algorithm for solving the distributional optimization.
4. R ELATED WORKS
The mean-ﬁeld description of SGD dynamics has been studied in several prior works for different information processing tasks.
Wang et al. Wang et al.
(2017) consider the problem of online learning for the principal component analysis (PCA), and analyze the scaling limits of different online learning algorithms based on the notion of ﬁnite exchangeability .
In their seminal papers, Montanari and co-authors Mei et al.
; Javanmard et al.
Mei et al.
(2019) consider the scaling limits of SGD for training a two-layer neural network, and characterize the related Mckean-Vlasov PDE for the limiting distribution of the empirical measure associated with the weights of the input layer.
Similar mean-ﬁeld type results for two-layer neural networks are also studied recently in Rotskoff & Vanden-Eijnden (2018); Sirignano & Spiliopoulos (2018).
Our work is also related to the unpublished work of Wang, et al.
Wang et al., which proposes a solvable model of GAN and analyzes the scaling limits.
However, our GAN model is different from Wang et al.
and is based on the notion of the kernel MMD.
Our work is also closely related to the recent work of Li, et al Li et al.
(2019) which proposes an implicit kernel learning method based on the following kernel deﬁnition
[Equation]
(25) where µ 0 is a user deﬁned base distribution, and
h ∈ H is a function that transforms the base distribution µ 0 into a distribution µ that provides a better kernel.
Therefore, the work of Li, et al Li et al.
(2019) implicitly optimizes the distribution of random features via transforming a random variable with a function.
In contrast, the proposed distributional optimization framework in this paper optimizes the distribution of random feature explicitly , via optimizing their empirical measures.
Perhaps more importantly from a practical perspective is the fact that our kernel learning approach does not require the user-deﬁned function class H .
Moreover, our particle SGD method in equation 15 obviates tuning hyper-parameters related to the implicit kernel learning method such as the gradient penalty factor and the variance constraint factor (denoted by λ GP and λ h , respectively, in Algorithm 1 of Li et al.
E MPIRICAL EVALUATION 5. S YNTHETIC DATA - SET
Due to the space limitation, the experiments on the synthetic data are deferred to Appendix A. viii
5. P ERFORMANCE ON BENCHMARK DATASETS
We evaluate our kernel learning approach on large-scale benchmark data-sets.
We train our MMD
GAN model on two distinct types of datasets, namely on MNIST LeCun et al.
(1998) and CIFARLeCun et al.
(1998), where the size of training instances are 60 × 10 3 and 50 × 10 3 , respectively.
All the generated samples are from a ﬁxed noise random vectors and are not singled out.
Implementation and hyper-parameters.
We implement Algorithm 1 as well as MMD GAN Li et al.
(2017) in Pytorch using NVIDIA Titan V100 32GB graphics processing units (GPUs).
The source code of Algorithm 1 is built upon the code of Li et al.
(2017), and retains the auto-encoder implementation.
In particular, we use a sequential training of the auto-encoder and kernel as explained in the Synthetic data in Section A of Supplementary.
For a fair comparison, our hyper-parameters are adjusted as in Li et al.
(2017), i.e. , the learning rate of 0.00005 is considered for RMSProp
Tieleman & Hinton (2012).
Moreover, the batch-size for training the generator and auto-encoder is n = 64 .
The learning rate of particle SGD is tuned to η = 10 .
Random Feature Maps.
To approximate the kernel, we use the the random feature model of
Rahimi and Recht Rahimi & Recht (2008; 2009), where ϕ ( x ; ξ ) = √ 2 cos( x T ξ + b ) .
Here b ∼
Uniform {− π, + π } is a random bias term.
Practical considerations.
When data-samples { V i } ∈ IR
d are high dimensional (as in CIFAR10), the particles ξ 1 , · · · , ξ N ∈ IR D , D
= d in SGD equation 15 are also high-dimensional.
To reduce the dimensionality of the particles, we apply an auto-encoder architecture similar to Li et al.
(2017), and train our kernel on top of learned embedded features.
More speciﬁcally, in our simulations, we train an auto-encoder where the dimensionality of the latent space is h = 10 for MNIST, and h = 128 (thus D = d = 128 ) for CIFAR-10.
Therefore, the particles ξ 1 , · · · , ξ N in subsequent kernel training phase have the dimension of D = 10 , and D = 128 , respectively.
Choice of the scaling parameter α .
There is a trade-off in the choice of α .
While for large values of α , the kernel is better able to separate data-samples from generated samples, in practice, it slows down the convergence of particle SGD.
This is due to the fact that the coupling between the particle dynamics in equation 15 decrease as α increase.
The scaling factor is set to be α = 1 in all the following experiments.
Qualitative comparison.
We now show that without the bandwidth tuning for Gaussian kernels and using the particle SGD to learn the kernel, we can attain better visual results on benchmark data-sets.
In Figure 1, we show the generated samples on CIFAR-10 and MNIST data-sets, using our Algorithm 1, MMD GAN Li et al.
(2017) with a mixed and homogeneous Gaussian RBF kernels, and GMMN Li et al.
Figure 1(a) shows the samples from Algorithm 1, Figure 1(b) shows the samples from MMD GAN
Li et al.
(2017) with a mixture RBF Gaussian kernel κ ( x , y )
= (cid:80)
k
=1 κ σ k ( x , y ) , where σ k ∈ { 1 , 2 , 4 , 8 , 16 } are the bandwidths of the Gaussian kernels that are ﬁne tuned and optimized.
We observe that our MMD GAN with automatic kernel learning visually attains similar results to MMD
GAN Li et al.
(2017) which requires manual tuning of the hyper-parameters.
In Figure 1(c), we show the MMD GAN result with a single kernel RBF Gaussian kernel whose bandwidth is manually tuned at σ = 16 .
Lastly, in Figure 1(d), we show the samples from GMMN Li et al.
(2015) which does not exploit an auto-encoder or kernel training.
Clearly, GMMN yield a poor results compared to other methods due to high dimensionality of features, as well as the lack of an efﬁcient method to train the kernel.
On MNIST data-set in Figure 1,(e)-(h), the difference between our method and MMD GAN Li et al.
(2017) is visually more pronounced.
We observe that without a manual tuning of the kernel bandwidth and by using the particle SGD equation 15 to optimize the kernel, we attain better generated images in Figure 1(e), compared to MMD GAN with mixed RBF Gaussian kernel and manual bandwidth tuning in Figure 1(f).
Moreover, using a single RBF Gaussian kernel yields a poor result regardless of the choice of its bandwidth.
The generated images from GMMN is also shown in
Figure 1(h).
ix
[Figure]
Quantitivative comparison.
To quantitatively measure the quality and diversity of generated samples, we compute the inception score (IS) Salimans et al.
(2016) as well as Fr ` echet Inception Distance (FID)
Heusel et al.
(2017) on CIFAR-10 images.
Intuitively, the inception score is used for
GANs to measure samples quality and diversity.
Accordingly, for generative models that are collapsed into a single mode of distribution, the inception score is relatively low.
The FID improves on
IS by actually comparing the statistics of generated samples to real samples, instead of evaluating generated samples independently.
In Table 1, we report the quantitative measures for different MMD GAN model using different scoring metric.
Note that in Table 1 lower FID scores and higher IS scores indicate a better performance.
We observe from Table 1 that our approach attain lower FID score, and higher IS score compared to MMD GAN with single Gaussian kernel (bandwidth σ = 16 ), and a mixture Gaussian kernel (bandwidths { 1 , 2 , 4 , 8 , 16 } ).
Method
FID (
[Table]
IS ( ↑ )
[Table] [Table]
A PPENDIX
We provide additional material to support the content presented in the paper.
This Appendix is organized as follows:
Appendix A .
We provide the results of our experiments on the synthetic data-set described in Section 5.1 of the main text.
Appendix B .
We provide the results of our experiments on the LSUN and CelebA data-sets.
Appendix C .
We present the proofs of the main theoretical results of Section 4 in the main text.
Appendix D .
We present the proofs of auxiliary lemmas used to support the proof of main results.
Appendix E .
We prove additional theoretical results regarding the so-called chaoticity of the particle SGD in equation 15.
A E XPERIMENTAL R ESULTS ON THE S YNTHETIC D ATA -S ET
The synthetic data-set we consider is as follows:
• The distribution of training data is P V = N ( 0 , (1 + λ ) I d × d ) , • The distribution of generated data is P W = N ( 0 , (1 − λ )
I d × d ) .
xiii [Figure] To reduce the dimensionality of data, we consider the embedding ι : IR d (cid:55)→ IR p , x (cid:55)→ ι ( x ) = Σ x , where Σ ∈ IR p × d and p < d .
In this case, the distribution of the embedded features are P X | Y
=+
=
N ( 0 , (1 + λ ) ΣΣ T ) , and P X | Y = − 1 = N ( 0 , (1 − λ ) ΣΣ T ) .
Note that λ ∈ [0 , 1] is a parameter that determines the separation of distributions.
In particular, the Kullback-Leibler divergece of the two multi-variate Gaussian distributions is controlled by λ ∈ [0 , 1] ,
[Equation]
In Figure 2, we show the distributions of i.i.d.
samples from the distributions P V and P W for different choices of variance parameter of λ = 0 .
1 , λ = 0 .
5 , and λ = 0 .
Notice that for larger λ the divergence is reduced and thus performing the two-sample test is more difﬁcult.
From Figure 2, we clearly observe that for large values of λ , the data-points from the two distributions P V and P W have a large overlap and conducting a statistical test to distinguish between these two distributions is more challenging.
A.0. K ERNEL L EARNING A PPROACH
Figure 4 depicts our two-phase kernel learning procedure which we also employed in our implementations of Algorithm 1 on benchmark data-sets of Section 5.2 in the main text.
The kernel learning approach consists of training the auto-encoder and the kernel optimization sequentially, i.e. ,
[Equation]
where the function class is deﬁned Q def = { ι ( z ) =
Σ z , Σ ∈ IR D
× d } , and ( K (cid:98) µ N ◦ ι )( x 1 , x 2 ) =
K (cid:98) µ N ( ι ( x 1 ) , ι ( x 2 )) .
Now, we consider a two-phase optimization procedure:
• Phase (I) : we ﬁx the kernel function, and optimize the auto-encoder to compute a covariance matrix Σ for dimensionality reduction • Phase (II) : we optimize the kernel based on the learned embedded features.
This two-phase procedure signiﬁcantly improves the computational complexity of SGD as it reduces the dimensionality of random feature samples ξ ∈ IR D , D (cid:28) d .
When the kernel function K is ﬁxed, optimizing the auto-encoder is equivalent to the kernel learning step of Li et al. xiv
[Figure]
A.0. S TATISTICAL H YPOTHESIS T ESTING WITH THE K ERNEL MMD
Let V 1 , · · · , V m ∼ i.i.d.
P V = N ( 0 , (1 + λ ) I
d × d ) , and W 1 , · · · , W n ∼ i.i.d.
P W = N ( 0 , (1 − λ )
I d × d ) .
Given these i.i.d.
samples, the statistical test T ( { V i } mi =1 , { W i } nj =1 ) :
V m ×W n → { 0 , 1 } is used to distinguish between these hypotheses:
• Null hypothesis H 0 : P V = P W (thus λ = 0 ), • Alternative hypothesis H 1 : P V (cid:54) =
P W (thus λ > 0 ).
To perform hypothesis testing via the kernel MMD, we require that H X is a universal RKHS, deﬁned on a compact metric space X .
Universality requires that the kernel K ( · , · ) be continuous and, H X be dense in C ( X ) .
Under these conditions, the following theorem establishes that the kernel MMD is indeed a metric:
Theorem A.1.
(M ETRIZABLITY OF THE RKHS) Let F denotes a unit ball in a universal RKHS
H X deﬁned on a compact metric space X with the associated continuous kernel K ( · , · ) .
Then, the kernel MMD is a metric in the sense that MMD K [ P V , P W ] =
if and only if P V = P W .
To design a test, let (cid:98) µ Nm ( ξ ) = 1 N (cid:80) N k =1 δ ( ξ − ξ km ) denotes the solution of SGD in equationfor solving the optimization problem.
Consider the following MMD estimator consisting of two xv
and an empirical function
[Equation]
Given the samples { V i } mi =1 and { W i } ni =1 , we design a test statistic as below
[Equation]
) where τ ∈ IR is a threshold.
Notice that the unbiased MMD estimator of equation 28 can be negative despite the fact that the population MMD is non-negative.
Consequently, negative values for the statistical threshold τ equation 29 are admissible.
In the following simulations, we only consider non-negative values for the threshold τ .
A Type I error is made when H 0 is rejected based on the observed samples, despite the null hypothesis having generated the data.
Conversely, a Type II error occurs when H 0 is accepted despite the alternative hypothesis H 1 being true.
The signiﬁcance level of a test is an upper bound on the probability of a Type I error: this is a design parameter of the test which must be set in advance, and is used to determine the threshold to which we compare the test statistic.
The power of a test is the probability of rejecting the null hypothesis H 0 when it is indeed incorrect.
In particular,
Power def = IP( reject {+H 
+}
| H 1 is true ) .
[Equation]
In this sense, the statistical power controls the probability of making Type II errors.
A.0. E MPIRICAL R ESULTS
In Figure 3, we show the evolution of the empirical measure µ Nm ( ξ ) of SGD particles by plotting the 2D histogram of the particles ξ 1 m , · · · , ξ Nm ∈ IR D at different iterations of SGD ( D = d ) .
Clearly, starting with a uniform distribution in 3(a), the empirical measure seemingly evolves into a Gaussian measure in Figure 3(d).
The evolution to a Gaussian distribution demonstrates that the RBF Gaussian kernel corresponding to a Gaussian distribution for the random features indeed provides a good kernel function for the underlying hypothesis test with Gaussian distributions.
In Figure 4, we evaluate the power of the test for 100 trials of hypothesis test using the test statistics of equation 29.
To obtain the result, we used an autoencoder to reduce the dimension from d =to p = 50 .
Clearly, for the trained kernel in Panel (a) of Figure 4, the threshold τ for which Power = 1 increases after learning the kernel via the two phase procedure described earlier.
In comparison, in Panel (b), we observe that training an auto-encoder only with a ﬁxed standard Gaussian kernel
K ( x , y )
=
e −(cid:107)
x − y (cid:107) 22 attains lower thresholds compared to our two-phase procedure.
In Panel (c), we demonstrate the case of a ﬁxed Gaussian kernel without an auto-encoder.
In this case, the threshold is signiﬁcantly lower due to the large dimensionality of the data.
From Figure 4, we also observe that interestingly, the phase transition in the statistical threshold τ is not sensitive to the parameter λ .
This phenomenon can be justiﬁed by the fact that the kernel learning indeed improves the MMD value more signiﬁcantly for smaller values of λ ( i.e. , more difﬁcult hypothesis testing problems) than larger values of λ .
See Figure 5.
xvi [Figure] [Figure] B E XPERIMENTAL R ESULTS ON LSUN B EDROOM AND C ELEB A D ATA - SETS
In this section we present additional simulations using CelebA Liu et al.
and LSUN bedroom
Yu et al.
(2015) data-sets.
The LSUN dataset of bedroom pictures resized to 64 × and the CelebA dataset of celebrity face images resized and cropped to 160 × 160.
The sample generated images are shown in Figure
The inception score for LSUN bedroom data set is 3 .
860 ± 0 .
0423 using a single Gaussian kernel with the bandwidth of σ = 16 , and 4 .
064 ± 0 .
061 using our proposed method in Algorithm 1.
C P ROOFS OF M AIN T HEORETICAL R ESULTS
Before we delve into proofs, we state the main assumptions underlying our theoretical results:
Assumptions :
( A . 1 )
The feature space X
= V ∪ W ⊂ IR d is compact with a ﬁnite diameter diam( X ) < ∞ , where V = support( P V and W = support( P W ) are the supports of the distributions P V and P W respectively. xvii
[Figure]
( A . 2 )
The feature maps are bounded and Lipchitz almost everywhere (a.e.)
ξ ∈ IR D .
In particular, sup x ∈X | ϕ ( x ; ξ ) | <
L 0 , sup x ∈X (cid:107)∇ ξ ϕ ( x ; ξ ) (cid:107) 2 ≤ L 1 , and sup ξ ∈ IR D (cid:107)∇ x ϕ ( x ; ξ ) (cid:107) <
L 2 .
Let L def = max { L 0 , L 1 , L 2 } < + ∞ .
( A . 3 ) Let (cid:98) µ N 0 ( ξ ) def = 1 N (cid:80)
Nk =1 δ ( ξ − ξ k 0 ) denotes the empirical measure for the initial particles ξ 10 , · · · , ξ N 0 .
We assume that (cid:98) µ N 0 ( ξ ) converges (weakly) to a deterministic measure µ 0 ∈
M (IR D ) .
Furthermore, we assume the limiting measure µ 0 is absolutely continuous w.r.t.
Lebesgue measure and has a compact support support( µ 0 ) =
Ξ ⊂ IR D .
Notation : We denote vectors by lower case bold letters, e.g. x = ( x 1 , · · · , x n ) ∈ IR n , and matrices by the upper case bold letters, e.g. , M =
[ M ij ] ∈ IR n × m .
The Frobenius norm of a matrix is denoted by (cid:107) M (cid:107) F = (cid:80) ni =1 (cid:80) mj
=1 | M ij | 2 .
Let IB r ( x ) def = { y ∈
IR
d : (cid:107)
y − x (cid:107) 2 ≤ r } denote the
Euclidean ball of radius r centered at x .
For a given metric space X , Let C b (IR d ) denote the space of bounded and continuous functions on X equipped with the usual supremum norm
[Equation]
Further, C k b ( X ) the space of all functions in C b ( X ) whose partial derivatives up to order k are bounded and continuous, and C kc ( X ) the space of functions whose partial derivatives up to order k are continuous with compact support. xviii
We denote the class of the integrable functions f with f ( t ) ≥ 0 a.e. , on 0 ≤ t ≤ T by L 1+ [0 , T ] .
Similarly, L ∞ + [0 , T ] will denote the essentially bounded functions with f ( t ) ≥ 0 almost everywhere.
For a given metric space X , we denote the Borel σ -algebra by B ( X ) .
For a Borel set B ∈ B ( X ) , the measure value of the set B with respect to the measure is given by µ ( B ) .
The space of ﬁnite non-negative measures deﬁned on X is denoted by M ( X ) .
The Dirac measure with the unit mass at x ∈ X is denoted by δ ( x ) .
For any measure µ ∈ M ( X ) and any bounded function f ∈ C b ( X ) , we deﬁne
[Equation]
The space M ( X ) is equipped with the weak topology, i.e. , a (random) sequence { µ N } N ∈
IN converges weakly to a deterministic measure µ ∈ M ( X ) if and only if (cid:104) µ N , f (cid:105) → (cid:104) µ, f (cid:105) for all f ∈ C b ( X ) .
We denote the weak convergence by µ Nt weakly → µ .
Notice that when X is Polish, then M ( X ) equipped with the weak topology is also Polish.
For a Polish space X , let D X
([0 , T ]) denotes the Skorokhod space of the c´adl´ag functions that take values in X deﬁned on [0 , T ] .
We assume that D X ([0 , T ]) is equipped with the Skorokhod’s J 1 -topology Billingsley (2013), which in that case D X ([0 , T ]) is also a Polish space.
We use asymptotic notations throughout the paper.
We use the standard asymptotic notation for sequences.
If a n and b n are positive sequences, then a n = O ( b n ) means that lim sup n →∞ a n /b n < ∞ , whereas a n = Ω( b n ) means that lim inf n →∞ a n /b n > 0 .
Furthermore, a n = (cid:101) O ( b n ) implies a n = O ( b n poly log( b n )) .
Moreover a n = o ( b n ) means that lim n →∞ a n /b n = 0 and a n = ω ( b n ) means that lim n →∞ a n /b n = ∞ .
Lastly, we have a n = Θ( b n ) if a n = O ( b n ) and a n = Ω( b n ) .
Finally, for positive a, b > 0 , denote a (cid:46) b if a/b is at most some universal constant.
Deﬁnition C.1.
(O RLICZ N ORM )
The Young-Orlicz modulus is a convex non-decreasing function ψ : IR + → IR +
such that ψ (0) =
and ψ ( x ) → ∞ when x → ∞ .
Accordingly, the Orlicz norm of an integrable random variable X with respect to the modulus ψ is deﬁned as
[Equation]
In the sequel, we consider the Orlicz modulus ψ ν ( x ) def = ( x ν ) − 1 .
Accordingly, the cases of (cid:107) · (cid:107) ψand (cid:107)·(cid:107) ψ 1 norms are called the sub-Gaussian and the sub-exponential norms and have the following alternative deﬁnitions:
Deﬁnition C.2.
(S UB -G AUSSIAN N ORM )
The sub-Gaussian norm of a random variable Z , denoted by (cid:107) Z (cid:107) ψ 2 , is deﬁned as
[Equation]
For a random vector Z ∈ IR n , its sub-Gaussian norm is deﬁned as follows
[Equation]
Deﬁnition C.3.
(S UB - EXPONENTIAL N ORM )
The sub-exponential norm of a random variable Z , denoted by (cid:107) Z (cid:107) ψ 1 , is deﬁned as follows
[Equation]
For a random vector Z ∈ IR n , its sub-exponential norm is deﬁned below
[Equation]
xix
C. P ROOF OF T HEOREM 4.
By the triangle inequality
[Equation]
we have that
[Equation]
where the terms A i , i = 1 , 2 , 3 , 4 are deﬁned as follows
[Equation]
In the sequel, we compute an upper bound for each term on the right hand side of equation 38:
Upper bound on A 1 :
First, notice that the squared kernel MMD loss in equation 4 can be characterized in terms of class labels and features deﬁned in Section 3.1 as follows
[Equation]
To see this equivalence
[Equation]
we ﬁrst rewrite the right hand side of equation 39 as follows
IE
[Equation]
Now, recall from Section 3.1 that P x | y
=+1 =
P V , and P x | y
=
− 1 =
P W by construction of the labels and random features.
Moreover, y, (cid:98)
y ∼ i .
i .
d .
Uniform {− 1 , +1 } , and thus IP { y = − 1 } = IP { y = +1 } = 12 .
Therefore, from equation 40, we derive
IE P ⊗ 2 y, x [ y (cid:98) yK
[Equation]
x ,
[Equation]
[Equation] K
[Equation]
] =
[Equation]
4MMD µ [ P V , P W ] .
For any given W ∈ W
[Equation]
we have that
[Equation]
xx where the error term is deﬁned using the random features
[Equation]
Now
[Equation]
we invoke the following strong duality theorem Gao & Kleywegt (2016):
Theorem C.4.
(S TRONG D UALITY FOR R OBUST O PTIMIZATION , (G AO & K LEYWEGT , 2016,
T HEOREM 1)) Consider the general metric space (Ξ , d )
[Equation]
and any normal distribution ν ∈ M (Ξ) , where M (Ξ) is the set of Borel probability measures on Ξ .
Then,
[Equation]
provided that Ψ is upper semi-continuous in ξ .
Under the strong duality of Theorem C.
[Equation]
we obtain that
[Equation]
In the sequel, let p = 2 .
The Moreau’s envelope Parikh & Boyd (2014) of a function f : X → IR is deﬁned as follows
[Equation]
where β > 0 is the regularization parameter.
When the function f is differentiable, the following lemma can be established:
Lemma C.5.
(M OREAU ’ S ENVELOPE OF D IFFERENTIABLE F UNCTIONS )
Suppose the function f : X → IR is differentiable.
Then, the Moreau’s envelope deﬁned in equation 44 has the following upper bound and lower bounds
[Equation]
In particular, when f is L f -Lipschitz, we have
[Equation]
The proof is presented in Appendix D.1.
Now, we return to Equation equation 43.
We leverage the lower bound on Moreau’s envelope in equation 45 of Lemma C.5 as follows
[Equation]
xxi
Let ζ
[Equation]
=
[Equation] [Equation]
) = arg sup
[Equation]
E n (1 − s ) ξ + s ζ (cid:107) 2 .
Then, applying the union bound in conjunction with Inequality equation 47 yields
[Equation]
Now
[Equation]
we state the following lemma:
Lemma C.6.
(T AIL B OUNDS FOR THE F INITE S AMPLE E STIMATION E RROR )
Consider the estimation error E n deﬁned in equation 41.
Then, the following statements hold:
• Z
=
(cid:107)∇ E n ( ξ ) (cid:107) 2 2 is a sub-exponential random variable with the Orlicz norm of (cid:107) Z (cid:107) ψ 1 ≤ 9 × 2 9 × L 4 nfor every ξ ∈ IR D .
Moreover,
[Equation]
• E n ( ξ ) is zero-mean sub-Gaussian random variable with the Orlicz norm of (cid:107) E n ( ξ ) (cid:107) ψ 2 ≤ 16 √ 3 L 4 n for every ξ ∈ IR D .
Moreover,
[Equation]
The proof of Lemma C.6 is presented in Appendix C.
[Equation]
Now
[Equation]
we leverage the probability bounds equation 49 and equation 50 of Lemma C.6 to upper bound the terms on the right hand side of equation 48 as below
[Equation]
where the last inequality comes from the basic inequality a + b ≤ 2 max { a, b } .
Therefore, with the probability of at least 1 − (cid:37)
[Equation]
we have that
[Equation]
for all W ∈ W .
Lemma C.7.
(D ISTANCE BETWEEN MINIMA OF A DJACENT F UNCTIONS ) Let Ψ( W ) :
W → IR and Φ( W ) :
W → IR .
Further, suppose (cid:107) Ψ( W ) − Φ( W ) (cid:107) ∞
≤
δ for some δ > 0 .
Then,
[Equation]
53) xxii
See Appendix D.4 for the proof.
Let Ψ( W ) def = sup µ ∈P (cid:92) MMD µ [ P V , P W ] , and Φ( W ) def = sup µ ∈P MMD µ [ P V , P W ] .
Then, from
Inequality equation 53, we have the following upper bound on A
[Equation]
with the probability of (at least) 1 − (cid:37) .
Upper bound on A 2 :
To establish the upper bound on A 2 , we recall that
[Equation]
x j ; ξ )] .
[Equation]
55b)
Therefore,
[Equation]
Here, the last inequality is due to Theorem C
[Equation]
4 and the following duality results hold
[Equation]
Now, in the sequel
[Equation]
we establish a uniform concentration result for the following function
[Equation]
Then, from equation 56 we have
[Equation]
We now closely follow the argument of Rahimi & Recht (2008) to establish a uniform concentration result with respect to the data points x , (cid:98) x ∈ X .
In particular, consider an (cid:15) -net cover of X ⊂ IR d .
xxiii
Then, we require N (cid:15) =
[Equation]
4diam
[Equation]
(cid:17) d balls of the radius (cid:15) > 0 , e.g. , see (Pollard, 1990, Lemma 4.1, Section 4).
Let Z = { z 1 , · · · , z N (cid:15) } ⊂ X denotes the center of the covering net.
Now, let ( ξ 10 , · · · , ξ k 0 , · · · , ξ N 0 ) ∈
IR N × D and ( ξ 10 , · · · , (cid:101) ξ k 0 , · · · , ξ N
) ∈
IR N
× D be two sequences that differs in the k -th coordinate for 1 ≤ k ≤ N .
Then,
[Equation]
Without loss of generality suppose M 12 λ − ϕ
( z i
[Equation]
· ) ϕ
[Equation]
z j
[Equation]
· ) ( ξ k
[Equation]
) ≥ M
[Equation]
− ϕ ( z i ; · ) ϕ
( z j ; · ) ( (cid:101) ξ k 0 ) .
Then,
M
[Equation] [Equation]
=
[Equation]
where (a) follows by letting ζ = ξ k 0 in the ﬁrst optimization problem, (b) follows by using the fact that − λ (cid:107) ζ − (cid:101) ξ k 0 (cid:107) 2 is non-positive for any ζ ∈ IR D and can be dropped
[Equation]
and (c) follows from
Assumption ( A . 2 )
[Equation]
Now
[Equation]
plugging the upper bound in equation 59 into equation 58 yields
[Equation]
From McDiarmid’s Martingale inequality McDiarmid (1989) and the union bound
[Equation]
we obtain that
[Equation] [Equation]
60) for all λ ≥ 0 .
Now, consider arbitrary points ( x , (cid:98) x )
∈
X × X .
Let the center of the balls containing those points be z
i , z j ∈ Z , i.e. , x ∈ IB ε ( z i ) and (cid:98) x ∈ IB ε ( z j ) for some z i , z j ∈ Z .
Then, by the triangle inequality, we have that
[Equation]
where L T =
L T ( ξ 10 , · · · , ξ N 0 ) def = sup x , (cid:98) x ∈X (cid:107)∇ x T λ ( x , (cid:98) x ) ( ξ 10 , · · · , ξ N 0 ) (cid:107) 2 is the Lipschitz constant of the mapping T .
Note that the Lipschitz constant L T is a random variable with respect to the random feature samples ξ 0 , · · · , ξ N .
Let ( x ∗ , (cid:98) x ∗ ) def = arg sup x , (cid:98) x ∈X (cid:107)∇ x T λ ( x , (cid:98) x ) ( ξ 10 , · · · , ξ N 0 ) (cid:107) 2 .
We compute an upper bound on the second moment of the random variable L T as follows
[Table]
[Equation]
We further proceed using the triangle inequality as well as the basic inequality ( a 1 + a 2 + · · · + a N ) 2 ≤ N ( a 2 1 + a
[Equation]
· · · + a 2 N ) ,
[Equation]
To proceed from equation 62, we leverage the following lemma:
Lemma C.8.
(M OREAU ’ S E NVELOP OF P ARAMETRIC F UNCTIONS ) Consider the parametric function f : X × Θ → IR and the associated Moreau’s envelope for a given
[Equation]
Furthermore, deﬁne the proximal operator as follows
[Equation]
Then, Moreau’s envelope has the following upper bound
[Equation]
The proof is presented in Appendix D
[Equation]
[Equation]
Equipped with Inequality equation 65 of Lemma C.8, we now compute an upper bound on the right hand side of equation 62 as follows
[Equation]
66) where the last inequality is due to ( A . 2 ) .
[Equation]
Invoking Markov’s inequality now yields
[Equation]
Now, using the union bound, for every arbitrary pair of data points ( x , (cid:98) x ) ∈ X
× X the following inequality holds
[Equation]
Following the proposal of Rahimi & Recht (2008), we select (cid:15) =
( κ 1 /κ 2 ) 1 d +2 , where κ 1 def = (4diam( X ))
d · e − 2 Nλδ 2 2 L 2 λ + L 4 and κ 2 def = (2 /δ )
2 L 4 .
Then,
IP
[Equation]
Thus, with the probability of at least 1 − (cid:37)
[Equation]
the following inequality holds
[Equation]
where W( · ) is the Lambert W -function.
Since W( x ) ≤ ln( x ) for x > e , we can rewrite the upper bound in terms of elementary functions
[Equation]
provided that N is sufﬁciently large and/or (cid:37) is sufﬁciently small so that 2 8 N diam 2 ( X ) (cid:37) ≥ e .
Plugging
Inequality equation 70 in equation 57 now results in the following inequality
[Equation]
for all W ∈ W .
Employingequation 53 from Lemma C.7 now yields the following upper bound
[Equation]
xxvi
Upper bound on A 3 :
Recall that the solution of the empirical risk function of equation 14 is denoted by
[Equation]
We also deﬁne the solution of the empirical kernel alignment as follows
[Equation]
Due to the optimality of the empirical measure (cid:98) µ N ∗ for the inner optimization in equation
[Equation]
the following inequality holds
[Equation]
Upon expansion of (cid:92) MMD α (cid:98) µ N (cid:5) (cid:2) P V , P (cid:99) W N ∗ (cid:3)
[Equation]
and after rearranging the terms in equation
[Equation]
we arrive at
[Equation]
where the last inequality is due to the fact that (cid:107) ϕ (cid:107) ∞ <
L by (A.1) .
Now
[Equation]
due to optimality of (cid:99) W N (cid:5) for the outer optimization problem in equation
[Equation]
we have
[Equation]
Putting together Inequalities equation 76 and equation 77 yields
[Equation]
Similarly, due to the optimality of the empirical measure (cid:98) µ N (cid:5) for the optimization in equation 74 we have that
[Equation]
xxvii
Combining equation 78 and equation 79 then yields
[Equation]
Upper bound on A 4 :
The upper bound on A 4 can be obtained exactly the same way as A 1 .
Speciﬁcally, from equationit follows directly that
[Equation]
Now, plugging the derived upper bounds in A 1 - A 4 in equation 38 and employing the union bound completes the proof.
[Equation]
C. P ROOF OF T HEOREM 4.
The proof has three main ingredients and follows the standard procedure in the literature, see, e.g. ,
Wang et al.
; Luo & Mattingly (2017).
In the ﬁrst step, we identify the mean-ﬁeld limit of the particle SGD in equation 15.
In the second step, we prove the convergence of the measured-valued process { ( µ Nt )
≤ t ≤ T } to the mean-ﬁeld solution by establishing the pre-compactness of Sokhorhod space.
Lastly, we prove the uniqueness of the mean-ﬁeld solution of the particle SGD.
Step 1-Identiﬁcation of the scaling limit : First, we identify the weak limit of converging subsequences via the action of the empirical measure (cid:98) µ Nm ( ξ ) = 1 N (cid:80) Nk
=1 δ ( ξ − ξ km ) on a test function f ∈ C 3 b (IR D ) .
In particular, we use the standard techniques of computing the scaling limits from
Luo & Mattingly (2017).
Recall that the action of an empirical measure on a bounded function is deﬁned as follows
[Equation]
We analyze the evolution of the empirical measure (cid:98) µ Nm via its action on a test function f ∈ C 3 b (IR D ) .
Using Taylor’s expansion, we obtain
[Equation]
where R Nm is a remainder term deﬁned as follows
[Equation]
xxviii where (cid:101) ξ k def
=
( (cid:101) ξ k (1) , · · · , (cid:101) ξ k ( p )) , and (cid:101) ξ k
( i ) ∈
[ ξ km ( i ) , ξ km +1 ( i )] , for i = 1 , 2 , · · · , p .
Plugging the difference term ( ξ km +1 − ξ km ) from the SGD equation in equation 15 results in
[Equation]
[Equation]
Now, we deﬁne the drift and Martingale terms as follows
[Equation]
respectively.
Using the deﬁnitions of D Nm and M Nm in equation 86a-equation 86b
[Equation]
we recast Equation equation 85 as follows
[Equation]
Nm
[Equation]
Summation over (cid:96) = 0 , 1 , 2 · · · , m − 1 and using the telescopic sum yields
[Equation]
We also deﬁne the following continuous embedding of the drift, martingale, and the remainder terms as follows
[Equation]
(cid:96)
[Equation]
(89c)
The scaled empirical measure µ Nt def = (cid:98) µ N (cid:98) Nt (cid:99) then can be written as follows
[Equation]
Since the drift process ( D Nt ) 0 ≤ t ≤ T is a piecewise c´adl´ag process, we have
[Equation]
91) where the functional R [ µ s ] is deﬁned as follows
[Equation]
xxix
Therefore, the expression in equation 90 can be rewritten as follows
[Equation]
In the following lemma, we prove that the remainder term sup 0 ≤ t ≤ T |R Nt | vanishes in probabilistic sense as the number of particles tends to inﬁnity N → ∞ :
Lemma C.9.
(L ARGE N -L IMIT OF THE R EMAINDER P ROCESS )
Consider the remainder process ( R Nt )
≤ t ≤ T deﬁned via scaling in equation 84-equation 89c.
Then, there exists a constant C 0 > 0 such that
[Equation]
and thus lim sup N →∞ sup 0 ≤ t
[Equation]
T |R Nt | = 0 .
Proof.
The proof is relegated to Appendix D.5.
We can also prove a similar result for the process deﬁned by the remainder term:
Lemma C.10.
(L ARGE N -L IMIT OF THE M ARTINGALE P ROCESS )
Consider the Martingale process ( M Nt ) 0 ≤ t ≤ T deﬁned via scaling in equation 86b-equation 89b.
Then, for some constant C 1 > 0 , the following inequality holds
[Equation]
In particular, with the probability of at least 1 − ρ , we have
[Equation]
(96) and thus lim sup N →∞ sup 0 ≤ t ≤ T |M Nt | = 0 almost surely.
Proof.
The proof is deferred to Appendix D.6.
Now, using the results of Lemmata C.10 and C.9 in conjunction with equation 93 yields the following mean-ﬁeld equation as N → ∞ ,
[Equation]
Notice that he mean-ﬁeld equation in equation 97 is in the weak form.
When the Lebesgue density p t ( ξ ) =
d µ t / d ξ exists, the McKean-Vlasov PDE in equation 22 can be readily obtained from equation 97.
Step 2: Pre-compactness of the Skorkhod space : To establish our results in this part of the proof, we need a deﬁnition and a theorem: xxx
Deﬁnition C.11.
(T IGHTNESS )
A set A of probability measures on a metric space S is tight if there exists a compact subset S
⊂ S such that
[Equation]
all
[Equation]
for all ε > 0 .
A sequence { X N } N ∈
IN of random elements of the metric space S is tight if there exists a compact subset S
⊂ S such that
[Equation]
for all ε > 0 , and all N ∈ IN .
Now, to show the tightness of the measured valued process ( µ Nt ) 0 ≤ t ≤ T , we must verify Jakubowski’s criterion (Jakubowski, 1986, Thm. 1):
Theorem C.12.
(J AKUBOWSKI ’ S CRITERION (J AKUBOWSKI , 1986, T HM . 1))
A sequence of measured-valued process { ( ζ Nt ) 0 ≤ t ≤ T } N ∈
IN
is tight in D M (IR D ) ([0 , T ]) if and only if the following two conditions are satisﬁed:
( J . 1 )
For each T > 0 and γ > 0 , there exists a compact set U T,γ such that
[Equation]
This condition is referred to as the compact-containment condition.
( J . 2 )
There exists a family H of real-valued functions H : M (IR D ) (cid:55)→
IR that separates points in M (IR D ) and is closed under addition such that for every H ∈ H , the sequence { ( H ( ξ Nt ))
≤ t ≤ T } N ∈ IN is tight in D IR
([0 , T ]) .
To establish ( J1 ) , we closely follow the proof of (Giesecke et al., 2013, Lemma 6.1.).
In particular, for each L > 0 , we deﬁne S L = [0 , B ] p .
Then, S B ⊂ IR p is compact, and for each t ≥ 0 , and
N ∈
IN , we have
[Equation]
where (a) follows from Markov’s inequality
[Equation]
and (b) follows from the upper bound on the norm of the particles in equation 177 of Appendix D. We now deﬁne the following set
[Equation]
xxxi
We let U T,γ = U B , where U B is the completion of the set U B .
By deﬁnition, U T,γ is a compact subset of M (IR D ) .
Now, we have
[Equation]
Now, since
[Equation]
this implies that for any γ > 0 , there exists a B > 0 , such that
[Equation]
This completes the proof of ( J.1 ) .
To verify (J.2) , we consider the following class of functions
H def =
{ H : ∃ f ∈ C 3 b (IR D ) such that H ( µ ) =
(cid:104) µ, f (cid:105) , ∀ µ ∈ M (IR D ) } .
By deﬁnition, every function H ∈ H is continuous with respect to the weak topology of M (IR D ) and further the class of functions H separate points in M (IR D ) and is closed under addition.
Now, we state the following sufﬁcient conditions to establish (J.2) .
The statement of the theorem is due to (Robert, 2013, Thm.
C.9):
Theorem C.13.
(T IGHTNESS IN D IR
([0 , T ]) , (R OBERT , 2013, T HM .
C.9))
A sequence { ( Z Nt ) 0 ≤ t ≤ T } N ∈
IN
is tight in D IR
([0 , T ]) iff for any δ > 0 , we have
( T . 1 )
There exists (cid:15) > 0 , such
that
[Equation]
for all N ∈ IN .
( T . 2 )
For any ρ > 0 , there exists σ > 0 such that
[Equation]
This completes the tightness proof of the of the laws of the measured-valued process { ( µ Nt )
≤ t ≤ T } N ∈ IN .
Now, we verify the condition (J.2) by showing that the sufﬁcient conditions (T.1) and (T.2) hold for function values { ( H ( µ Nt ))
≤ t ≤ T } N ∈
IN , where H ∈ H and H is deﬁned in
Eq.
equation 108.
Now, condition ( T . 1 ) is readily veriﬁed since
[Equation]
113) xxxii where in the last step, we used the fact that f ∈ C 3 b (IR D ) , and hence, (cid:107) f (cid:107) ∞ ≤ b .
Thus, IP( H ( µ N 0 ) ≥ b ) = 0 for all N ∈ IN , and the condition ( T . 1 ) is satisﬁed.
Now, consider the condition (T.2) .
From
Equation equation 93, and with 0 ≤ t 1 < t 2 ≤ T we have
[Equation]
To bound the ﬁrst term, recall the deﬁnition of R [ µ s ] from equation 92.
The following chain of inequalities holds,
[Equation]
Let I : IR D → IR , I ( ξ ) =
denotes the identity function.
Notice that (cid:104)
[Equation]
From equation
[Equation]
we proceed as follows
[Equation]
where the last inequality is due to (A.1) .
Therefore,
[Equation]
where s 0 def = 2 η α
( L 2 + α )
L 2 C 1 .
Consider the middle term of equation 114.
Using the deﬁnition of the martingale term in equation 89b
[Equation]
we obtain that
[Equation]
(cid:12)(cid:12) (cid:12)(cid:12)(cid:12)(cid:12) (cid:98)
[Equation]
(cid:99) (cid:88) (cid:96) = (cid:98)
Nt 1 (cid:99)
M N(cid:96) (cid:12)(cid:12) (cid:12)(cid:12)(cid:12)(cid:12) .
In Equation of Section D, we have proved the following concentration bound
[Equation] [Equation]
Now, recall the alternative deﬁnition of the sub-Gaussian random variables:
Deﬁnition C.14.
(S UB -G AUSSIAN R ANDOM V ARIABLES B OUCHERON ET AL .
A random variable X is σ 2 -sub-Gaussian if
[Equation]
120) xxxiii
We enumerate a few standard consequences of sub-Gaussianity Boucheron et al.
If X i are independent and σ 2 i -sub-Gaussian, then (cid:80) ni
=1 X i is (cid:80) ni
=1 σ 2 i -sub-Gaussian.
Moreover, X is σ 2 -sub-Gaussian if and only if
[Equation]
Now, it is clear from equation 119 andthat M Nm is sub-Gaussian random variable with a zero mean, and with the parameter σ 2 m = 4 mL 4 η 2 C 21 ( L 2 + α )
N 2 α.
Therefore, (cid:80) (cid:98)
Nt 2 (cid:99) (cid:96) =
(cid:98)
Nt 1 (cid:99) M N(cid:96) is sub-Gaussian with the parameter σ 2 ( t 1 , t 2 ) def =
2 L 4 η 2 C 21 ( L 2 + α )
N 2 α( (cid:98)
Nt 1 (cid:99) − (cid:98)
Nt 2 (cid:99) + 1)( (cid:98)
Nt 1 (cid:99) + (cid:98)
Nt 2 (cid:99) ) .
Consequently, from Inequality equation 118 and the concentration inequality in equation 121, we have
IP
[Equation]
(cid:12) (cid:12)(cid:12)(cid:12)(cid:12)(cid:12) (cid:98)
Nt 2 (cid:99) (cid:88) (cid:96) =
(cid:98)
Nt 1 (cid:99)
M N(cid:96)
[Equation]
where ( t ∗ 1 , t ∗ 2 ) def = arg sup t 1 ,
[Equation]
2 ≤
[Equation]
, |
[Equation] [Equation]
≤
[Equation]
We ﬁrst compute a bound for the last term of equation 114 using the deﬁnition of the scaled term
R Nt from equation 89c.
We have
[Equation]
=
[Equation] [Equation]
where (a) follows from the upper bound in equation 178 of Section D, and in (b) we deﬁne s 1 def =
C
N ( ηL 2 + ( L 4 /α )) .
xxxiv
Putting together equation 117, equation 122, and equation 126, we conclude from Inequality equa- tion 114 that
[Equation]
Therefore, condition (T.2) is also satisﬁed.
Since the sufﬁcient conditions (T.1) and (T.2) are satisﬁed, the condition (J.2) is satisﬁed.
This completes the tightness proof of the measured-valued sequence { µ Nt } N ∈ IN .
Now, we prove its convergence to a mean-ﬁeld solution ( µ ∗ t )
≤ t ≤ T .
Theorem C.15.
(P ROKHOROV ’ S THEOREM P ROKHOROV (1956))
A subset of probability measures on a complete separable metric space is tight if and only if it is pre-compact.
According to Theorem C.15, the tightness of the Skorkhod Space D M (IR D ) ([0 , T ]) implies its precompactness which in turn implies the existence of a converging sub-sequence { ( µ Nt ) 0 ≤ t ≤ T } N k of { µ Nt } N ∈ IN .
Notice that { ( µ Nt ) 0 ≤ t ≤ T } N k is a stochastic process deﬁned on the Skorkhod space.
Therefore, let π N k denotes the law of the converging sub-sequence { ( µ Nt )
≤ t ≤ T } N k .
By deﬁnition, π N k is an element of the measure space M
( D [0 ,T ] ( M (IR D ))) .
In the sequel, we closely follow the argument of (Wang et al., 2017, Proposition 4) to show that the limiting measure π ∞ is a Dirac’s delta function concentrated at a mean-ﬁeld solution µ ∗ t ∈ D [0 ,T ] ( M (IR D )) .
We deﬁne the following functional
[Equation]
t
[Equation]
We compute the expectation of the functional F t with respect to π N k .
We then have
[Equation]
Now, from Equation equation
[Equation]
we have that
[Equation]
Plugging equation 129 in equation 128 gives
IE π Nk [ F
[Equation]
IE
[Equation]
F t [
[Equation] [Equation]
where the last equality is due to the bounds in equation 94 and equation 95 of Lemmata C.9 and
C.
[Equation]
respectively.
Taking the limit of N → ∞ from equation 130 yields
[Equation]
xxxv
It can be shown that the functional F t [ · ] is continuous and bounded.
Therefore, due the weak convergence of the sequence { π N k } N k ∈ IN to π ∞ , equation 131 implies that
[Equation]
Since the identity equation 132 holds for all bounded test functions f ∈ C 3 b (IR D ) and for all t ∈ (0 , T ] , it follows that π ∞ is a Dirac’s delta function concentrated at a solution ( µ ∗ t )
≤ t ≤ T of the mean-ﬁeld equation.
Step 3: Uniqueness of a mean-ﬁeld solution : Before we establish the uniqueness result we make two remarks:
First, we make it clear that from the compact-containment condition (J.1) of Jakubowski’s criterion in Theorem C.12, the support of the measured-valued process ( µ Nt ) 0 ≤ t ≤ T = (
[Equation] µ N
[Equation]
Nt
[Equation]
) 0 ≤ t ≤ T is compact for all 0 ≤ t ≤ T .
Moreover, in Step 2 of the proof, we established that the measure valued process ( µ Nt )
≤ t ≤ T converges weakly to a mean-ﬁeld solution as the number of particles tends to inﬁnity ( i.e. , N → ∞ ).
Thus, all the possible solutions of the mean-ﬁeld equation also have compact supports.
Let (cid:98) Ξ ⊂ IR D denotes a compact set containing the supports of all such solutions at 0 ≤ t ≤ T .
In the sequel, it sufﬁces to establish the uniqueness of the mean-ﬁeld solution for the test functions with a compact domain, i.e. , let f ∈ C 3 b ( (cid:98) Ξ) .
Second, for all bounded continuous test functions f ∈ C 3 b ( (cid:98) Ξ) , the operator f → (cid:104) µ t , f (cid:105) is a linear operator with µ t (IR D )
= 1 .
Hence, from Riesz-Markov-Kakutani representation theorem Rudin (1987); Varadarajan (1958) by assuming µ t ∈ M (IR D ) , existence of unique operator implies f (cid:55)→ (cid:104) f, µ t (cid:105) implies the existence of the unique probability measure µ t .
Now, we equip the measure space M (IR D ) with the following norm
[Equation]
Given an initial measure µ 0 , we next prove that there exists at most one mean-ﬁeld model solution by showing that there exists at most one real valued process (cid:104) µ t , f (cid:105) corresponding to the mean-ﬁeld model.
Suppose ( µ ∗ , 1 t ) 0 ≤ t ≤ T , ( µ ∗ , 2 t ) 0 ≤ t ≤ T are two solutions satisfying the mean-ﬁeld equations equation 97 with the initial distributions µ 10 , µ 20 ∈ M
(IR D ) , respectively.
For any test function f ∈ C 3 b ( (cid:98) Ξ) we have that
[Equation]
We bound the ﬁrst term on the right side of Equation equation 134 as follows
[Equation]
[Equation]
where used the deﬁnition of the norm (cid:107) · (cid:107) on the measure space M (IR D ) from equation 133.
Furthermore, let
[Equation]
where in the last inequality, we used the fact that (cid:107)∇ f ( ξ ) (cid:107) ≤ C 1 since the test function is threetimes continuously differentiable f ∈ C 3 b ( (cid:98) Ξ) on a compact support.
Similarly, we have
[Equation]
Putting together the inequalities in equation
[Equation]
equation
[Equation]
and equation 138 yield
[Equation]
The above inequality holds for all bounded functions f ∈ C 3 b ( (cid:98) Ξ) .
Thus, by taking the supremum with respect to f we obtain
[Equation]
Now, we employ the following result which generalizes Gronewall’s inequality when higher order terms are involved:
Lemma C.16.
(E XTENDED G RONEWALL ’ S INEQUALITY , (W EBB , 2018, T HM 2.1.))
Let p ∈ IN and suppose that for a.e. t ∈ [0 , T ] , u ∈ L ∞ + [0 , T ] satisﬁes [Equation]
[Equation]
where c 0 ∈ L ∞ [0 , T ] is non-decreasing, and c j ∈ L 1+ [0 , T ] for j ∈ { 1 , · · · , p + 1 } .
Then, if
[Equation]
It follows that for a.e. t ∈ [0 , T ]
[Equation]
144) xxxvii
We now apply the extended Gronewall’s Inequality equation 142 with p = 1 , c 0 ( t )
=
b (cid:107) µ 10 − µ 20 (cid:107) , 0 ≤ t ≤ T , c 1 ( t ) =
ηL 2 C 1 , 0 ≤ t ≤ T , c 2 ( t )
=
ηL 4 C 1 α , 0 ≤ t ≤ T , and u s = (cid:107) µ ∗ , 1 s − µ ∗ , 2 s (cid:107) .
In this case, it is easy to see that M 1 = 2 bηTL 4 C 1 α .
Consequently, from equation 140 and equation 144, we obtain that
[Equation]
Thus, starting from an initial measure µ 10 = µ 20 = µ 0 , there exists at most one solution for the mean-ﬁeld model equations equation 97.
[Equation]
C. P ROOF OF C OROLLARY 4.2.
To establish the proof, we recall from equation 88 that
[Equation]
146) for all f ∈ C b (IR D ) .
Recall the deﬁnition of the total variation distance TV( · , · ) on a metric space ( X , d ) .
[Equation]
The total variation distance admits the following variational form
[Equation]
Now, using the variation form and by taking the supremum of equation 146 with respect to the functions from the function class F c def = { f ∈ C 1 / 2 (IR D ) } , we obtain the following upper bound on the TV distance
[Equation]
Based on the upper bound equation 178 on the remainder term, we have
[Equation]
for some constant C 0 > 0 .
Moreover, from the concentration inequality equation 189, we also have that with the probability of at least 1 − δ , the following inequality holds
[Equation]
Lastly, recall the deﬁnition of the drift term in equation 86a.
By carrying out a similar bounding method leading to equation 116, it can be shown that
[Equation]
By plugging equation 150, equation 151, and equation 152 into equation 149, we derive that
[Equation]
with the probability of 1 − δ
[Equation]
We now leverage the following lemma: xxxviii
Lemma C.17.
(B OUNDED E QUIVALENCE OF THE W ASSERSTEIN AND T OTAL V ARIATION D ISTANCES , S INGH & P ´ OCZOS (2018))
Suppose ( X , d ) is a metric space, and suppose µ and ν are
Borel probability measures on X with countable support; i.e., there exists a countable set X (cid:48) ⊆
X such that µ ( X (cid:48) )
=
ν ( X (cid:48) )
= 1 .
Then, for any p ≥ 1 , we have
[Equation]
where Diam( X (cid:48) )
def = sup x,y ∈X (cid:48)
d ( x, y ) , and
[Equation]
( X (cid:48) )
def = inf x (cid:54)
=
y ∈X (cid:48)
[Equation]
x
[Equation]
Consider the metric space (IR D , (cid:107) · (cid:107) 2 ) .
Note that the empirical measures (cid:98) µ Nm , (cid:98) µ N 0 have a countable support X (cid:48)
= { ξ km } Nk =1 ∪ { ξ k 0 } Nk =1 ⊂ IR D .
Therefore, using the upper bounds in equation 154 of
Lemma C.17 and 153, we conclude that when the step-size is of the order η
= O (cid:18)
[Equation]
(cid:19) , (155) then W p ( (cid:98) µ N m , (cid:98) µ N 0 ) ≤ R for all m ∈ [0 , NT ] ∩ IN .
(cid:4)
D P ROOFS OF A UXILIARY R ESULTS D. P ROOF OF L EMMA C.
The upper bound follows trivially by letting x = y in the optimization problem equation 44.
Now, consider the lower bound.
Deﬁne the function g : [0 , 1] → IR , t (cid:55)→ g ( t ) =
f ( y + t ( x − y )) .
Then, when f is differentiable, we have g (cid:48) ( t ) =
(cid:104) x − y , ∇ f ( y + t ( x − y ))
(cid:105) .
In addition, g (0) =
f ( y ) , and g (1) =
f ( x ) .
Based on the basic identity g (1) =
g (0)
+ (cid:82) 1 0 g (cid:48) ( s )
d s , we derive
[Equation]
where the last step is due to the Cauchy-Schwarz inequality.
Using Inequality equation 156 yields the following lower bound on Moreau’s envelope
[Equation]
where (a) is due to Jensen’s inequality
[Equation]
D. P ROOF OF L EMMA C.
Let u ∈
IR
d denotes an arbitrary unit vector (cid:107) u (cid:107) 2 = 1 .
From the deﬁnition of the gradient of a function, we have that
[Equation]
where D u [ M f ( · ; θ ) ( x )] is the directional derivative
[Equation]
We now have
[Equation]
where the inequality in (a) follows by letting y
=
Prox f ( · ; θ ) ( x ) in the optimization problem
[Equation]
Now, recall that
[Equation]
Therefore,
[Equation]
Substitution of equation 161 in equation 160 yields
[Equation]
Hence, D u [ M f ( · ; θ ) ( x )] ≤ (cid:104)∇ θ f
(Prox f ( · ; θ ) ( x ); θ ) , u (cid:105) .
From equation 158 and by using Cauchy- Schwarz inequality, we compute the following bound on the inner product of the gradient with the unit vectors
[Equation]
∈
[Equation] [Equation]
Since the preceding upper bound holds for all the unit vectors u ∈ IR d , we let u =
[Equation]
to get Inequality equation 65.
[Equation]
D. P ROOF OF L EMMA C.
Let z ∈ S d − 1 denotes an arbitrary vector on the sphere.
Deﬁne the random variable
Q z (cid:16)
[Equation]
xl
Clearly, IE P x ,y [ Q z ] = 0 .
Now
[Equation]
let ( (cid:98) y m , (cid:98) x m ) ∈
Y × X , 1 ≤ m ≤ n .
By repeated application of the triangle inequality
[Equation]
we obtain that
[Equation]
where the last inequality is due to assumption ( A.2 ) and the fact that (cid:107) z (cid:107) 2 = 1 for z ∈ S d − 1 .
In particular
[Equation]
to derive Inequality equation
[Equation]
we employed the following upper bounds
[Equation]
Using McDiarmid Martingale’s inequality McDiarmid (1989) then gives us
[Equation]
for x ≥ 0 .
Now, for every p ∈ IN , the 2 p -th moment of the random variable Q z is given by
IE
[Equation] [Equation]
where (a) is due to the concentration bound in equation 166.
Now Therefore,
IE (cid:2)(cid:0) Q 2 z (( y 1 , x 1 ) , · · · , ( y n , x n )) /γ 2 (cid:1)(cid:3) =
[Equation]
L 4 /nγ
[Equation]
− 1 .
For γ = 16 √ 3 L 4 /n , we obtain IE (cid:2)(cid:0) Q 2 z (( y 1 , x 1 ) , · · · , ( y n , x n ))
/γ
(cid:1)(cid:3) ≤ 2 .
Therefore, (cid:107) Q z (cid:107) ψ
= (cid:107)(cid:104) z , ∇ E n ( ξ ) (cid:105)(cid:107) ψ 2 ≤ 16 √ 3 L 4 /n for all z ∈ S n − 1 and ξ ∈ IR D .
Consequently, by the deﬁnition xli of the sub-Gaussian random vector in equation 35 of Deﬁnition C.2, we have (cid:107)∇ E n ( ξ ) (cid:107) ψ 2 ≤ 16 √ 3 L 4 /n for every ξ ∈ IR D .
We invoke the following lemma proved by the ﬁrst author in (Khuzani & Li, 2017, Lemma 16):
Lemma D.1.
(T HE O RLICZ N ORM OF THE S QUARED V ECTOR N ORMS , (K HUZANI & L I , 2017,
L EMMA 16)) Consider the zero-mean random vector Z satisfying (cid:107) Z (cid:107) ψ ν ≤ β for every ν ≥ 0 .
Then, (cid:107)(cid:107) Z (cid:107) 22 (cid:107) ψ ν 2 ≤ 2 · 3 2 ν · β 2 .
Using Lemma D.1, we now have that (cid:107)(cid:107)∇ E n ( ξ ) (cid:107) 22 (cid:107) ψ 1 ≤ 4608 L 4 /n 2 for every ξ ∈ IR D .
Applying the exponential Chebyshev’s inequality with β = 4608 L 4 /n 2 yields
[Equation]
where (a) follows by Jensen’s inequality
[Equation]
and (b) follows from the fact that
[Equation]
due to Deﬁnition C
[Equation]
Therefore,
[Equation]
It now remains to compute an upper bound on the expectation
[Equation]
But this readily follows from equation 167 by letting p = 1 and z = ∇ E n ((1 − s )
ξ
+ s ζ ∗ )
(cid:107)∇ E n ((1 − s ) ξ
+ s ζ ∗ ) (cid:107) 2 as follows
IE x ,y [ (cid:107)∇
[Equation] [Equation]
Plugging the expectation upper bound of equation 170 into equation 169 completes the proof of the ﬁrst part of Lemma C.6.
The second part of Lemma C.6 follows by a similar approach and we thus omit its proof.
(cid:4)
D. P ROOF OF L EMMA C.
Let W ∗ def =
arg min W ∈W Ψ( W ) and W (cid:5)
def = arg min
W ∈W Φ( W ) .
Then, since | Ψ( W ) − Φ( W ) | ≤ δ for all W ∈ W
[Equation]
we have that
[Equation]
Therefore,
[Equation]
Similarly
[Equation]
it can be shown that
[Equation]
Combining equation 172 and equation 173 yields the desired inequality.
[Equation]
D. P ROOF OF L EMMA C.
We recall the expression of the remainder term { R Nm } 0 ≤ m ≤ NT from equation 84.
For each 0 ≤ m ≤ NT , N ∈
IN
[Equation]
we can bound the absolute value of the remainder term as follows
[Equation]
Next
[Equation]
we characterize a bound on the difference term (cid:107) ξ km +1 − ξ km (cid:107) 2 .
To attain this goal
[Equation]
we use the iterations of the particle SGD in Equation equation 15.
We have that
[Equation]
where in (a)
[Equation]
we used the fact that (cid:107) ϕ (cid:107) ∞ <
L and (cid:107)∇ ξ ϕ ( x , ξ ) (cid:107) 2 <
L due to (A.1) [Equation] and y m , (cid:101) y m ∈ {− 1 , 1 } .
Plugging the last inequality in equation 175 yields
[Equation]
xliii We next compute an upper bound on the Frobenious norm of the Hessian matrix ∇ 2 f ( (cid:101) ξ k ) .
To this end, we ﬁrst show that there exists a compact set C ⊂ IR D such that ξ km ∈ C for all k = 1 , 2 , · · · , N and all m ∈ [0 , NT ] ∩ IN .
For each k = 1 , 2 , · · · , N , from Inequality equation 175 we obtain that
[Equation]
Now, (cid:107) ξ k 0 (cid:107) 2 < c 0 for some constant c 0 > 0 since the initial samples ξ 10 , · · · , ξ N 0 are drawn from the measure µ 0 whose support support( µ 0 ) =
Ξ is assumed to be compact by (A.3) .
From upper bound in equation 177, it thus follows that (cid:107) ξ km (cid:107) 2 < C for some constant C > 0 , for all m ∈ [0 , NT ] ∩ IN .
Now, recall that (cid:101)
ξ k = ( (cid:101) ξ k (1) , · · · , (cid:101) ξ k ( p )) , where (cid:101) ξ k
( i ) ∈
[ ξ km ( i ) , ξ km +1 ( i )] , i = 1 , 2 , · · · , m + 1 , for i = 1 , 2 , · · · , p .
Therefore, (cid:101) ξ k ∈ C .
Since all the test function f ∈ C 3 b (IR 3 ) are three-times continuously differentiable, it follows that there exists a constant C 0 def = C 0 ( T ) > 0 such that
[Equation]
f ( (cid:101) ξ ) (cid:107) F < C 0 .
From Inequality equation 176, it follows that
[Equation] [Equation]
] ∩
[Equation]
Now, recall the deﬁnition of the scaled term R Nt from equation 89c.
Using the Inequality equation 178 as well as the deﬁnition of R Nt , we obtain
[Equation]
D. P ROOF OF L EMMA C.
Let F m − 1 =
σ
( x k , y k ) 0 ≤ k ≤ m − 1 , ( (cid:101) x k , (cid:101) y k ) 0 ≤ k ≤ m − 1 ) denotes the σ -algebra generated by the samples up to time m − 1 .
We deﬁne F − 1 def = ∅ .
Further, deﬁne the following random variable
[Equation] [Equation]
Notice that 1 N IE[∆ Nm |F m − 1 ] =
D Nm .
We now rewrite the martingale term in equation 86b in term of ∆ Nm ,
[Equation]
with M N 0 = 0 .
By construction of M Nm in equation 181, it is a Martingale IE[ M Nm |F m − 1 ] =
[Equation]
Nm [Equation]
We now prove that M Nm has also bounded difference.
To do so, we deﬁne the shorthand notations
[Equation]
xliv
Then, we compute
[Equation]
For the difference terms
[Equation]
we derive that
[Equation]
where the last step follows from the fact that (cid:107) ϕ (cid:107) ∞ ≤ L due to (A.1) .
Similarly
[Equation]
we obtain that
[Equation]
where (a) and (c) follows from ( A . 1 )
[Equation]
and (b) follows from the Cauchy-Schwarz inequality.
From
Inequality equation 177 and the ensuing disucssion in Appendix D.5, we recall that (cid:107) ξ km (cid:107) 2 < C for some constant and for all m ∈ [0 , NT ] ∩ IN
[Equation]
and k = 1 , 2 , · · · , N .
For the two times continuously test function f ∈ C 3 b (IR D ) , it then follows that |∇ f ( ξ km ) (cid:107) 2 ≤ C 1 for some constant C 1 > 0 .
The following bound can now be computed from equation 186,
[Equation]
Plugging the upper bounds on | a Nm | and | b Nm | from equation 185-equation 187 into equation 184 we obtain that
[Equation]
Thus
[Equation]
( M Nm ) m ∈ [0 ,NT ] ∩ IN is a Martingale process with bounded difference.
From the Azuma- Hoeffding inequality it follows that
[Equation]
xlv
Therefore, since M Nt = M N (cid:98) Nt (cid:99) , we have
[Equation]
Then,
[Equation]
where the inequality follows from equation
[Equation]
By Doob’s
Martingale inequality Doob (1953)
[Equation]
the following inequality holds
[Equation]
In particular, with the probability of at least 1 − ρ , we have
[Equation]
E C HAOTICITY AND PROPAGATION OF CHAOS IN PARTICLE SGD
In this appendix, we establish the so called ‘propagation of chaos’ property of particle SGD.
We now establish the so called ‘propagation of chaos’ property of particle SGD.
At a high level, the propagation of chaos means that when the number of samples { ξ k } Nk =1 tends to inﬁnity ( N → + ∞ ) , their dynamics are decoupled.
Deﬁnition E.1.
(E XCHANGABLITY )
Let ν be a probability measure on a Polish space S and.
For
N ∈
IN , we say that ν ⊗ N is an exchangeable probability measure on the product space S n if it is invariant under the permutation π def
=
( π (1) , · · · , π ( N )) of indices.
In particular,
[Equation]
for all Borel subsets B ∈ B ( S n ) .
An interpretation of the exchangablity condition equation 195 can be provided via De Finetti’s representation theorem which states that the joint distribution of an inﬁnitely exchangeable sequence of random variables is as if a random parameter were drawn from some distribution and then the random variables in question were independent and identically distributed, conditioned on that parameter.
Next, we review the mathematical deﬁnition of chaoticity, as well as the propagation of chaos in the product measure spaces:
Deﬁnition E.2.
(C HAOTICITY )
Suppose ν ⊗ N is exchangeable.
Then, the sequence { ν ⊗ N } N ∈ IN is ν -chaotic if, for any natural number (cid:96) ∈
IN and any test function f 1 , f 2 , · · · , f k ∈ C 2 b ( S ) , we have
[Equation]
xlvi
According to equation 196 of Deﬁnition E.2, a sequence of probability measures on the product spaces S is ν -chaotic if, for ﬁxed k the joint probability measures for the ﬁrst k coordinates tend to the product measure
ν (d s 1 ) ν
(d s 2 ) · · · ν (d s k ) =
ν ⊗ k on S k .
If the measures ν ⊗ N are thought of as giving the joint distribution of N particles residing in the space S , then { ν ⊗ N } is ν -chaotic if k particles out of N become more and more independent as N tends to inﬁnity, and each particles distribution tends to ν .
A sequence of symmetric probability measures on S N is chaotic if it is ν -chaotic for some probability measure ν on S .
If a Markov process on S N begins in a random state with the distribution ν ⊗ N , the distribution of the state after t seconds of Markovian random motion can be expressed in terms of the transition function K N for the Markov process.
The distribution at time t > 0 is the probability measure
U Nt ν ⊗ N is deﬁned by the kernel
[Equation]
Deﬁnition E.3.
(P ROPOGATION OF C HAOS )
A sequence functions
[Equation]
) whose N -th term is a Markov transition function on S N that satisﬁes the permutation condition
[Equation]
(199) propagates chaos if whenever { ν ⊗ N } N ∈
IN is chaotic, so is { U Nt } for any t ≥ 0 , where U Nt is deﬁned in equation 197.
We note that for ﬁnite systems size N ,the states of the particles are not independent of each other.
However, as we prove in the following result, in the limiting system N → + ∞ , the particles are mutually independent.
This phenomena is known as the propagation of chaos ( a.k.a.
asymptotic independence):
Theorem E.4.
(C HAOTICITY IN P ARTICLE SGD) Consider Assumptions ( A . 1 ) − ( A . 3 ) .
Furthermore, suppose that { ξ k 0 } 1 ≤ k ≤ N ∼ i.i.d.
µ 0 is exchangable in the sense that the joint law is invariant under the permutation of indices.
Then, at each time instant t ∈ (0 , T ] , the scaled empirical measure µ Nt ∈ M (IR D ) deﬁned via scaling
[Equation]
is µ ∗ t -chaotic, where µ ∗ t is mean-ﬁeld solution of equation 97.
Proof .
To establish the proof, it sufﬁces to show that for every integer (cid:96) ∈ IN , and for all the test functions f 1 , · · · , f k ∈ C 3 b (IR D ) , we have
[Equation]
Using the triangle inequality, we now have that
[Equation]
IE
[Equation]
− (cid:96) (cid:89) k = (cid:104) µ ∗ t , f k (cid:105) (cid:12)(cid:12)(cid:12)(cid:12)(cid:12) ≤ (cid:12) (cid:12) (cid:12) (cid:12)(cid:12) IE (cid:34) (cid:96) (cid:89) k = (cid:104) (cid:98) µ Nt , f k (cid:105) (cid:35) − (cid:96) (cid:89) k
= (cid:104) µ ∗ t , f k (cid:105) (cid:12) (cid:12) (cid:12) (cid:12)(cid:12) + (cid:12) (cid:12) (cid:12) (cid:12)(cid:12) IE (cid:34) (cid:96) (cid:89) k
= (cid:104) (cid:98) µ Nt , f k (cid:105) (cid:35) − IE (cid:34) (cid:96) (cid:89) k
= f k ( ξ k (cid:98) Nt (cid:99) )
(cid:35)(cid:12) (cid:12) (cid:12) (cid:12)(cid:12) .
For the ﬁrst term on the right side of equation 202 we have
[Table]
[Equation]
where (a) is by Jensen’s inequality, (b) is by Fatou’s lemma, (c) follows from the basic inequality (cid:12) (cid:12) (cid:12) (cid:81) N i
=
a i − (cid:81) N i =
b
i (cid:12) (cid:12)(cid:12) ≤ (cid:80) N i =1 | a i − b i | for | a i | , | b i | ≤ 1 , i = 1 , 2 , · · · , N , as well as the fact that (cid:104) µ ∗ t , f k (cid:105) ≤ b and (cid:104) (cid:98) µ Nt , f k (cid:105) ≤ b for all k = 1 , 2 , · · · , N due to the boundedness of the test functions f 1 , · · · , f (cid:96) ∈ C 3 b (IR D ) , and (d) follows from the weak convergence (cid:98) µ Nt weakly → µ ∗ t to the mean-ﬁeld solution equation 97.
Now, consider the second term on the right hand side of equation 202.
Due to the exchangeability of the initial states ( ξ k 0 ) 1 ≤ k ≤ N , the law of the random variables ( ξ k 0 ) 1 ≤ k ≤ N is also exchangeable.
Therefore, we obtain that
IE
[Equation]
where Π( (cid:96), N ) is the set of all permutations of (cid:96) numbers selected from { 1 , 2 , · · · , N } .
Notice that the right hand side of equation 204 is the symmetrized version of the left hand side equation 205.
Further, by the deﬁnition of the empirical measure (cid:98) µ Nt we obtain that
[Equation]
Therefore, subtracting equation 204 and equation 205 yields
[Equation]
Hence,
[Equation]
Combining equation 203-equation 208 yields the desired result.
[Equation]
xlviii