S UFFICIENT
AND
D ISENTANGLED
R EPRESENTATION
L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
We propose a novel representation learning approach called sufﬁcient and disentangled representation learning (SDRL).
With SDRL, we seek a data representation that maps the input data to a lower-dimensional space with two properties: sufﬁciency and disentanglement.
First, the representation is sufﬁcient in the sense that the original input data is conditionally independent of the response or label given the representation.
Second, the representation is maximally disentangled with mutually independent components and is rotation invariant in distribution.
We show that such a representation always exists under mild conditions on the input data distribution based on optimal transport theory.
We formulate an objective function characterizing conditional independence and disentanglement.
This objective function is then used to train a sufﬁcient and disentangled representation with deep neural networks.
We provide strong statistical guarantees for the learned representation by establishing an upper bound on the excess error of the objective function and show that it reaches the nonparametric minimax rate under mild conditions.
We also validate the proposed method via numerical experiments and real data analysis.
I NTRODUCTION
Representation learning is a fundamental problem in machine learning and artiﬁcial intelligence (Bengio et al., 2013).
Certain deep neural networks are capable of learning effective data representation automatically and achieve impressive prediction results.
For example, convolutional neural networks, which can encode the basic characteristics of visual observations directly into the network architecture, is able to learn effective representations of image data (LeCun et al., 1989).
Such representations in turn can be subsequently used for constructing classiﬁers with outstanding performance.
Convolutional neural networks learn data representation with a simple structure that captures the essential information through the convolution operator.
However, in other application domains, optimizing the standard cross-entropy and least squares loss functions do not guarantee that the learned representations enjoy any desired properties (Alain & Bengio, 2016).
Therefore, it is imperative to develop general principles and approaches for constructing effective feature representations for supervised deep learning.
There is a growing literature on representation learning in the context deep neural network modeling.
Several authors studied the internal mechanism of supervised deep learning from the perspective of information theory (Tishby & Zaslavsky, 2015; Shwartz-Ziv & Tishby, 2017; Saxe et al., 2019), where they showed that training a deep neural network that optimizes the information bottleneck (Tishby et al., 2000) is a trade-off between the representation and prediction at each layer.
To make the information bottleneck idea more practical, deep variational approximation of information bottleneck (VIB) is considered in Alemi et al.
Information theoretic objectives describing conditional independence such as mutual information are utilized as loss functions to train a representation-learning function, i.e., an encoder in the unsupervised setting (Hjelm et al., 2018;
Oord et al., 2018; Tschannen et al., 2019; Locatello et al., 2019; Srinivas et al., 2020).
There are several interesting extensions of variational autoencoder (VAE) (Kingma & Welling, 2013) in the form of
VAE plus a regularizer for unsupervised representation learning, including beta-VAE (Higgins et al., 2017), Annealed-VAE (Burgess et al., 2018), factor-VAE (Kim & Mnih, 2018), beta-TC-VAE (Chen et al., 2018), DIP-VAE (Kumar et al., 2018).
The idea of using a latent variable model has also been
We propose a sufﬁcient and disentangled representation learning (SDRL) approach in the context of supervised learning.
Our approach is inspired by the seminal works on sufﬁcient dimension reduction (Li, 1991; Cook, 1998) and representation learning (Bengio et al., 2013; Achille & Soatto, 2018).
With SDRL, we seek a data representation with two characteristics.
First, it must be sufﬁcient in the sense that it preserves all the information about the supervised learning task.
This is accomplished by formulating a conditional independence principle, which stipulates that, given the representation, the original input data does not contain any additional information about the response variable.
This is motivated by and closely related to the fundamental concept of sufﬁcient statistics in parametric statistical models (Fisher, 1922).
Second, the representation should have a statistical structure as simple as possible.
We require that the components of the representation to be disentangled in the sense that they are statistically independent (Achille & Soatto, 2018).
We also require that the representation is rotation invariant in distribution.
Such an invariance property is an important characteristic in many visual and morphological data (Takacs et al., 2013; Dieleman et al., 2015).
Interestingly, these two requirements imply that that distribution of the representation is necessarily Gaussian, by a characterization of multivariate Gaussian distribution due to James C.
Maxwell (Maxwell, 1860).
Under mild conditions, there always exists a sufﬁcient representation that is disentangled and rotation invariant in distribution based on optimal transport theory (Brenier, 1991; McCann, 1995; Villani, 2008).
In the rest of the paper, we ﬁrst discuss the motivation and the theoretical framework for learning a sufﬁcient and disentangled representation map (SDRM).
This framework leads to the formulation of an objective function based on the conditional independence principle and a metric for disentanglement and invariance adopted in this work.
We estimate the target SDRM based on the sample version of the objective function using deep neural networks and develop an efﬁcient algorithm for training the SDRM.
We establish an upper error bound on the measure of conditional independence and disentanglement and show that it reaches the nonparametric minimax rate under mild regularity conditions.
This result provides strong statistical guarantees for the proposed method.
We validate the proposed SDRL via numerical experiments and real data examples.
S UFFICIENT AND DISENTANGLED REPRESENTATION
Consider a pair of random vectors (
x , y ) ∈
R p × R q , where x is a vector of input variables and y is a vector of response variables or labels.
Our goal is to ﬁnd a sufﬁcient and disentangled representation of x .
Sufﬁciency.
We say that a measurable map g : R p
→ R d with d ≤ p is a sufﬁcient representation of x if
[Equation]
that is, y and x are conditionally independent given g ( x ) .
This condition holds if and only if the conditional distribution of y given x and that of y given g ( x ) are equal.
Therefore, the information in x about y is completely encoded by g ( x ) .
Such a g always exists, since if we simply take g ( x ) = x , then (1) holds trivially.
This formulation is a direct generalization of the basic condition in sufﬁcient dimension reduction (Li, 1991; Cook, 1998), where it is assumed g ( x ) =
B T x with
B ∈ R p ×
d belonging to the Stiefel manifold, i.e., B T B = I d .
Denote the class of sufﬁcient representations satisfying (1) by
[Equation]
measurable and satisﬁes y x | g ( x ) } .
We refer to F as a Fisher class because of its close connection with the concept of sufﬁcient statistics (Fisher, 1922; Cook, 2007).
For an injective measurable transformation T : R d → R d and g ∈ F ,
T ◦ g ( x ) is also sufﬁcient by the basic property of conditional probability.
Therefore, the Fisher class F is invariant in the sense that T
[Equation]
F = F , provided T is injective, where T ◦ F = { T ◦ g : g ∈ F} .
An important class of transformations is the class of afﬁne transformations, T ◦ g =
Ag + b , where A is a d × d nonsingular matrix and b ∈ R d .
If we take A to be an orthogonal matrix and b = 0 , then F is rotation invariant.
Such an invariance property is appealing because it enables us to ﬁnd members of F with additional desirable properties based on appropriate transformations.
Disentanglement.
Disentangling is based on the notion that some latent causes underlie data generation process: although the observed data are typically high-dimensional, complex and noisy, the latent factors are low-dimensional, independent and have a relatively simple statistical structure.
This notion is clearly in line with the concept of sufﬁciency deﬁned in (1), where it posits that a lower-dimensional data representation contains all the relevant statistical information.
Our approach does not use a latent variable model, we start from the functions of the input data that are sufﬁcient representations in the Fisher class F .
There is a range of deﬁnitions of disentangling (Achille & Soatto, 2018; Higgins et al., 2018; Eastwood & Williams, 2018; Ridgeway & Mozer, 2018; Do & Tran, 2020).
Several metrics have been proposed for the evaluation of disentangling.
We seek a representation in F that is maximally disentangled in the sense that the total correlation is minimized, where the total correlation is deﬁned as the KL divergence between the joint distribution of g ( x ) and the product of the marginal distributions of its components (Watanabe, 1960).
If the components of g ( x ) are independent, then g ( x ) is (totally) disentangled.
Conversely, a zero total correlation implies independence of the components by the property of KL-divergence.
We view sufﬁciency as the most basic property of a data representation and focus on the disentangled representations among those that are sufﬁcient.
To further simplify the statistical structure of a representation g , we also require it to be rotation invariant in distribution, that is, Qg ( x ) =
g ( x ) in distribution for any orthogonal matrix Q ∈ R d × d .
The Fisher class F is rotation invariant in terms of conditional independence, but not all its members are rotation invariant in distribution.
By the
Maxwell characterization of the Gaussian distributions (Maxwell, 1860; Bartlett, 1934; Bryc, 1995;
Gyenis, 2017), a random vector of dimension two or more with independent components is rotation invariant in distribution if and only if it is Gaussian with zero mean and a spherical covariance matrix.
Therefore, after absorbing the scaling factor, for a sufﬁcient representation map to be disentangled and rotation invariant, it is necessarily distributed as N d ( 0 , I d ) .
Let M be the Maxwell class of functions
g : R d → R d , where g ( x ) is disentangled and rotation invariant in distribution.
By the
Maxwell characterization, we can write
[Equation]
Now our problem becomes that of ﬁnding a representation in F ∩ M , the intersection of the Fisher class and the Maxwell class.
The ﬁrst question to ask is whether such a representation exists.
The following result from optimal transport theory provides an afﬁrmative answer and guarantees that
F ∩ M is nonempty under mild conditions (Brenier, 1991; McCann, 1995; Villani, 2008).
Lemma 2.1.
Let µ be a probability measure on R d .
Suppose it has ﬁnite second moment and is absolutely continuous with respect to the standard Gaussian measure, denoted by γ d .
Then it admits a unique optimal transportation map T : R d → R d such that T # µ = γ d ≡ N ( 0 , I d ) , where T # µ denotes the pushforward distribution of µ under T .
Moreover, T is injective µ -almost everywhere.
Denote the law of a random vector z by µ z .
Lemma 2.1 implies that, for any g ∈ F with E (cid:107) g
( x ) (cid:107) 2 < ∞ and µ g ( x ) absolutely continuous with respect to γ d , there exists a map T ∗ transforming the distribution of g ( x ) to N ( 0 , I d ) .
Therefore, R ∗ :=
T ∗ ◦ g ∈ F ∩ M , that is,
[Equation]
i.e., R ∗ is a sufﬁcient and disentangled representation map (SDRM).
O BJECTIVE FUNCTION FOR SDRL
The above discussions lay the theoretical foundation for formulating an objective function that can be used for constructing a SDRM R ∗ satisfying (3), or equivalently, R ∗ ∈ F ∩ M .
Let V be a measure of dependence between random variables x and y with the following properties:
(a) V [ x , y ] ≥ 0 with V [ x , y ]
=
if and only if x y ; (b) V [ x , y ] ≥ V
[ R ( x ) , y
] for all measurable
function R ; (c) V [ x , y ]
=
V [ R ∗ ( x ) , y ] if and only if R ∗ ∈ F .
The properties (a)-(c) imply that
[Equation]
We use a divergence measure D to quantify the difference between µ R ( x ) and γ d , as long as this measure satisﬁes the condition D ( µ R ( x ) (cid:107) γ d ) ≥ 0 for all measurable function R and D ( µ R ( x ) (cid:107) γ d ) =
if and only if R ∈ M .
Then the problem of ﬁnding an R ∗ ∈ F ∩ M can be expressed as a constrained minimization problem:
[Equation]
to
[Equation]
Its Lagrangian form is
[Equation]
where λ ≥ 0 is a tuning parameter.
We show in Lemma 4.1 below that any R ∗ satisfying (3) is a minimizer of L ( R ) .
Therefore, we can train a SDRM by minimizing the empirical version of L ( R ) .
There are several options for V with the properties (a)-(c) described above.
For example, we can take
V to be the mutual information V
[ R ( x ) , y ]
= I ( R ( x ); y ) .
However, in addition to the estimation of the SDRM R , this choice requires the estimation of the density ratio between p ( y , R ( x )) and p ( y ) p ( R ( x )) , which is not an easy task.
We can also use the conditional covariance operators on reproducing kernel Hilbert spaces (Fukumizu et al., 2009).
To be speciﬁc, in this work we use the distance covariance (Sz ´ ekely et al., 2007) of y and R ( x ) , which has an elegant U -statistic expression, does not involve additional unknown quantities and is easy to compute.
For the divergnce measure of two distributions, we use the f -divergence (Ali & Silvey, 1966), which includes the KLdivergence as a special case.
Other divergence mesures such as the Wasserstein distance (Villani, 2008) can also be used.
L EARNING SUFFICIENT AND DISENTANGLED REPRESENTATION
We ﬁrst describe some essentials about distance covariance and f -divergence.
Distance covariance.
We ﬁrst recall the concept of distance covariance (Sz´ekely et al., 2007), which characterizes the dependence of two random variables.
Let i be the imaginary unit ( − 1) 1 / 2 .
For any t ∈ R d and s ∈ R m , let ψ z ( t ) =
E [exp i t T z ] , ψ y ( s ) =
E [exp i s T y ] , and ψ z , y ( t , s ) =
E [exp i ( t T z + s T y ) ] be the characteristic functions of random vectors z ∈ R d , y ∈ R q , and the pair ( z , y ) , respectively.
The squared distance covariance V [ z , y ] is deﬁned as
[Equation]
c
[Equation]
Given n i.i.d copies { z i , y i } ni =1 of ( z , y ) , an unbiased estimator of V is the empirical distance covariance (cid:98) V n , which can be elegantly expressed as a U -statistic
(Huo & Sz´ekely, 2016)
[Equation]
where h is the kernel deﬁned by
[Equation]
f-divergence.
Let µ and γ be two probability measures on R d .
The f -divergence (Ali & Silvey, 1966) between µ and γ with µ (cid:28) γ is deﬁned as D f ( µ (cid:107) γ ) =
(cid:82) R d f ( d µ d γ )d γ, where f : R + → R is a differentiable convex function satisfying f (1) = 0 .
Let f ∗ be the Fenchel conjugate of f (Rockafellar, 1970), deﬁned as f ∗ ( t ) =
sup x ∈ R { tx − f ( x ) } , t ∈ R .
The f -divergence admits the following variational formulation (Keziou, 2003;
Nguyen et al., 2010; Nowozin et al., 2016).
Lemma 4.1.
[Equation]
where the maximum is attained at D ( z )
=
f (cid:48)
( d µ d γ ( z )) .
Commonly used divergence measures include the Kullback-Leibler (KL) divergence, the JensenShanon (JS) divergence and the χ 2 -divergence.
Learning SDRM.
We are now ready to formulate an empirical objective function for learning SDRM R ∗ .
Let R ∈ M , where M is the Maxwell class deﬁned in (2).
By the variational formulation (6), we can write the population version of the objective function (4) as
[Equation]
This expression is convenient since we can simply replace the expectations by the corresponding empirical averages.
Theorem 4.2.
We have R ∗ ∈ arg min R ∈M
L ( R ) provided (3) holds.
According to Theorem 4.2, it is natural to estimate R ∗ based on the empirical version of the objective function (7) when a random sample { ( x i , y i ) } n i =1 is available.
We estimate R ∗ using deep neural networks.
We employ two networks as follows:
• Representer network R θ : This network is used for training R ∗ .
Let R be the set of such neural networks R θ : R p
→ R d . • Discriminator network D φ : This network is used to pushforward the distribution of the estimator of R ∗ to N ( 0 , I d ) .
This ensures that the components of the representation to be disentangled and rotation invariant.
Similarly, denote D as the set of such neural networks D φ : R d → R .
Let { w i } ni =1 be n i.i.d random vectors drawn from γ d .
The estimated SDRM is deﬁned by
[Equation]
where (cid:98) V n [ R θ ( x ) , y ] is an unbiased and consistent estimator of λ V [ R θ ( x ) , y ] as deﬁned in (5) based on { ( R θ ( x i ) , y i ) , i = 1 , . .
. , n } and
[Equation]
Statistical guarantee.
Since a SDRM R ∗ is only identiﬁable up to orthogonal transforms under the constraint that R ∗ ( x ) ∼ N (0 , I d ) , no consistency results for (cid:98) R θ itself can be obtained.
But this is not a ﬂaw of the proposed method.
Indeed, the most important statistical guarantee of the learned
R ∗ is that the objective of conditional independence and disentanglement is achieved.
Therefore, we establish an upper bound on the excess risk L ( (cid:98) R θ )
−L ( R ∗ ) of the deep nonparametric estimator (cid:98) R θ in (8).
We make the following assumptions.
(A1)
Suppose supp( µ x ) ⊆
[ − B 1 , B 1 ] p and R ∗ is Lipschitz continuous on [ − B 1 , B 1 ] p with Lipschitz constant L 1 .
(A2)
For R ∈ M , we assume r ( z ) =
d µ R ( x ) d γ d ( z ) is Lipschitz continuous on [ − B 1 , B 1 ] p with Lipschitz constant L 2 , and 0 < c 1 ≤ r ( z ) ≤ c 2 .
Denote B
[Equation]
The speciﬁcations of the network parameters, including depth, width, size and the supremum norm over the domains of the representer R θ and the discriminator D φ are given in Appendix B.
Theorem 4.3.
Let λ
=
O (1) .
Suppose conditions (A1)-(A2) hold and set the network parameters according to (i)-(ii).
Then
[Equation]
where C is a constant that depends on B 1 , B 2 and B 3 but not on n, q, p and d .
The proof of this theorem is given in Appendix B.
The result established in Theorem 4.3 provides strong statistical guarantees for the proposed method.
The rate n − 2 / (2+ p ) matches the minimax nonparametric estimation rate for Lipschitz class contained in R p of functions (Stone, 1982; Tsybakov, 2008).
Up to a log n factor, the rate (log n ) n − 2 / 2+ d matches the minimax rate of nonparametric estimation of Lipschitz densities via GANs (Singh et al., 2018; Liang, 2018).
C OMPUTATION
We can update θ and φ alternately as in training GANs (Goodfellow et al., 2014).
However, this approach suffers from the instability issues.
In our implementation, we utilize the more stable particle method based on gradient ﬂow (Gao et al., 2019; 2020).
The key idea is to ﬁnd a sequence of nonlinear but simpler residual maps, say T ( z )
=
z + s v ( z ) , pushing the samples from µ R θ ( x ) to the target distribution γ d along a velocity ﬁelds v ( z ) =
−∇ f (cid:48)
( r ( z )) that most decreases the f -divergence D f ( ·|| γ d ) at µ R θ ( x ) .
The residual maps can be estimated via deep density-ratio estimators, which take the form T ( z ) =
z + s (cid:98) v ( z ) , z ∈ R d , where s is a step size and (cid:98) v ( z ) =
− f (cid:48)(cid:48)
(ˆ r ( z )) ∇ ˆ r ( z ) .
Here ˆ r ( z ) is an estimated density ratio of the density of R θ ( x ) at the current value of θ over the density of the reference distribution.
We use T to transform z i =
R θ ( x i ) , i = 1 , . . . , n into Gaussian samples.
Once this is done, we update θ via minimizing the loss − (cid:98) V n [ R θ ( x ) , y ]+ λ (cid:80) ni
=1 (cid:107)
R θ ( x i ) −
z i (cid:107) 2 /n.
We describe the algorithm below.
• Input { x i , y i } ni =1 .
Tuning parameters: s, λ, d .
Sample { w i } ni
=1 ∼ γ d . • Outer loop for θ – Inner loop (particle method) ∗ Let z i
= R θ ( x i ) , i = 1 , 2 , ..., n .
∗ Solve (cid:98) D φ ∈ arg min Q φ (cid:80)
ni =1 1 n (cid:0) log(1 + exp D φ ( z i ) )
+ log(1 + exp − D φ ( w i ) ) (cid:1) .
∗ Deﬁne the residual map T ( z ) =
z − sf (cid:48)(cid:48)
(ˆ r ( z )) ∇ ˆ r ( z ) with ˆ r ( z ) =
exp − (cid:98) D
φ ( z ) .
∗ Update the particles z i
= T ( z i ) , i = 1 , 2 , ..., n .
– End inner loop – Update θ via minimizing − (cid:98) V n [ R θ ( x ) , y ] + λ (cid:80) ni
=1 (cid:107)
R θ ( x i ) −
z i (cid:107) 2 /n using SGD.
• End outer loop
E XPERIMENTS
We evaluate the proposed SDRL with the KL-divergence using both simulated and real data.
Details on the network structures and hyperparameters are included in
Appendix A.
Our experiments were conducted on Nvidia DGX Station workstation using a single Tesla V100 GPU unit.
The PyTorch code of SDRL is available at https://github.com/anonymous/SDRL .
6. S IMULATED DATA
In this subsection, we evaluate SDRL on simulated regression and classiﬁcation problems.
Regression.
We generate 5 , 000 data points from two models.
Model A: y = x 1 [0 .
+ ( x 2 + 1 . 5) 2 ] − 1 + (1 + x 2 ) 2 + σ ε , where x ∼ N ( 0 , I 4 ) ; Model B: y = sin 2 ( π x 1 + 1) + σ ε , where x ∼ Uniform([0 , 1] 4 ) .
In both models, ε ∼ N ( 0 , I 4 ) .
We use a 3-layer network with
ReLU activation for R θ and a single hidden layer ReLU network for D φ .
We compare SDRL with two prominent sufﬁcient dimension reduction methods: sliced inverse regression (SIR) (Li, 1991) and sliced average variance estimation (SAVE) (Cook & Weisberg, 1991).
We ﬁt a linear model with the learned features and the response variable, and report the prediction errors in Table 1.
We see that SDRL outperforms SIR and SAVE in terms of prediction error.
Classiﬁcation.
We visualize the learned features of SDRL on two simulated datasets.
We ﬁrst generate (1) 2-dimensional concentric circles from two classes as in Figure 1 (a); (2) 3-dimensional
Gaussian data from six classes as in Figure 1 (e).
In each dataset, we generate 5,000 data points for each class.
We next map the data into 100-dimensional space using matrices with entries i.i.d
[Table]
Unifrom([0 , 1]) .
Finally, we apply SDRL to these 100 -dimensional datasets to learn 2 -dimensional features.
We use a 10-layer dense convolutional network (DenseNet) (Huang et al., 2017) as R θ and a 4-layer network as D φ .
We display the evolutions of the learned 2-dimensional features by SDRL in Figure 1.
For ease of visualization, we push all the distributions onto the uniform distribution on the unit circle, which is done by normalizing the standard Gaussian random vectors to length one.
Clearly, the learned features for different classes in the examples are well disentangled.
[Figure]
6. R EAL DATASETS
Regression.
We use a benchmark YearPredictionMSD dataset to demonstrate the prediction performance of SDRL ( https://archive.ics.uci.edu/ml/datasets/YearPredictionMSD ) .
This dataset has 515,345 observations with 90 predictors.
The problem is to predict the year of song release.
We randomly split the data into ﬁve parts for cross validated evaluation of the prediction performance.
We employ a 3-layer network for both D φ and R θ .
A linear regression model is ﬁtted using the learned representations and the response.
The mean prediction errors and their standard errors based on SDRL, principal component analysis (PCA), sparse principal component analysis (SPCA) and ordinary least squares (OLS) regression with original data are reported in Table 2.
SDRL outperforms PCA, SPCA and OLS in terms prediction accuracy.
[Table]
Classiﬁcation.
We benchmark the classiﬁcation performance of SDRL using MNIST (LeCun et al., 2010), FashionMNIST (Xiao et al., 2017), and CIFAR-10 (Krizhevsky et al., 2009) against alternative methods including convolutional networks (CN) and distance correlation autoencoder (dCorAE) (Wang et al., 2018).
With CN, we use the feature extractor by dropping the cross entropy layer of the DenseNet trained for classiﬁcation.
The MNIST and FashionMNIST datasets consist of 60k and 10k grayscale images with 28 × 28 pixels for training and testing, respectively, while the CIFARdataset contains 50k and 10k colored images with 32 × 32 pixels for training and testing, respectively.
The representer network R θ has 20 layers for MNIST data and 100 layers for CIFAR-10 data.
The discriminator network D φ is a 4-layer network.
Finally, we use the k -nearest neighbor ( k = 5 ) classiﬁer on the learned features for all methods.
The classiﬁcation accuracies are reported in Table 3.
We can see that the classiﬁcation accuracies of SDRL are comparable with those of CN and dCorAE.
We also calculate the estimated distance correlation (DC) between the learned features and their labels.
Figure 2 shows the DC values FashionMNIST and CIFAR-10 data.
SDRL achieves higher DC values.
[Table]
[Figure]
for FashionMNIST (FMNIST) and CIFAR10 data.
C ONCLUSION AND FUTURE WORK
In this work, we formulate a framework for sufﬁcient and disentangled representation learning and construct an objective function characterizing conditional independence and disentanglement.
This enables us to learn a representation with the desired properties empirically.
We provide statistical guarantees for the learned representation by deriving an upper bound on the excess risk of the objective function.
There are several questions that deserve further study.
First, we can adopt different measures of conditional independence including mutual information and conditional covariance operators on reproducing kernel Hilbert spaces (Fukumizu et al., 2009).
We can also use other divergence measures such as the Wasserstein distance in the objective function.
Finally, Lemma 2.1 suggests that the intersection of the Fisher class F and the Maxwell class M can still be large, and there can be many statistically equivalent representations in F ∩ M .
We can make further reduction of F ∩ M by imposing additional constraints, for example, certain minimal properties, sparsity, and robustness against noise perturbation.
A A PPENDIX : E XPERIMENTAL DETAILS A. S IMULATION STUDIES
The values of the hyper-parameters for the simulated experiments are given in Table A1, where λ is the penalty parameter, d is the dimension of the SDRM, n is the mini-batch size in SGD, T 1 is the number of inner loops to push forward particles z i , T 2 is the number of outer loops for training R θ , and s is the step size to update particles.
For the regression models, the neural network architectures are shown in Table A
As shown in Table A a multilayer perceptron (MLP) is utilized for the neural structure D φ in the classiﬁcation problem.
The detailed architecture of 10-layer dense convolutional network (DenseNet) (Huang et al., 2017; Amos & Kolter) deployed for R θ is shown in Table A4.
For all the settings, we adopted the Adam (Kingma & Ba, 2014) optimizer with an initial learning rate of 0.
and weight decay of 0.0001.
[Table] [Table]
[Table] [Table]
Convolution
[Table]
Dense Block(cid:20) BN, 1 × 1 Conv
BN, 3 × 3 Conv (cid:21) ×
[Table]
×
Transition Layer
BN, ReLU, 2 × 2 Average Pool, 1 × 1 Conv 30 × 10 ×
Dense Block(cid:20) BN, 1 × 1 Conv
BN, 3 × 3 Conv (cid:21) ×18 × 10 ×
Transition Layer
BN, ReLU, 2 × 2 Average Pool, 1 × 1 Conv 15 × 5 ×
Dense Block(cid:20) BN, 1 × 1 Conv
BN, 3 × 3 Conv (cid:21) ×27 × 5 ×
Pooling
BN, ReLU, 5 × 5 Average Pool, Reshape
Fully connected
[Table]
[Table] [Table]
Layer
Linear, LeakyReLU Linear, LeakyReLU
Layer
Linear, LeakyReLU Linear, LeakyReLU
Layer
Linear
Linear d
A. R EAL DATASETS
Regression:
In the regression problems, hyper-parameters are presented in Table A5.
The Adam optimizer with an initial learning rate of 0.001 and weight decay of 0.0001 is adopted.
The MLP architectures of D φ and R θ for the YearPredictionMSD data are shown in Table A6.
Classiﬁcation: We again use Adam as the SGD optimizers for both D φ and R θ .
Speciﬁcally, learning rate of 0.001 and weight decay of 0.0001 are used for D φ in all datasets and for R θ on MNIST (LeCun et al., 2010).
We customized the SGD optimizers with momentum at 0.9, weight decay at 0.0001, and learning rate ρ in Table A8 for FashionMNIST (Xiao et al., 2017) and CIFAR-
(Krizhevsky et al., 2012), MLP architectures of the discriminator network D φ for MNIST, FashionMNIST and CIFAR-10 are given in Table A3.
The 20-layer DenseNet networks shown in Table A were utlized for R θ on the MNIST dataset, while the 100-layer DenseNet networks shown in Table
A10 and A11 are ﬁtted for R θ on FashionMNIST and CIFAR-10.
[Table] [Table]
[Table]
Dense Block(cid:20) BN, 1 × 1 Conv
BN, 3 × 3 Conv (cid:21) ×48 × 28 ×
Transition Layer
BN, ReLU, 2 × 2 Average Pool, 1 × 1 Conv 24 × 14 ×
Dense Block(cid:20) BN, 1 × 1 Conv
BN, 3 × 3 Conv (cid:21) ×48 × 14 ×
Transition Layer
BN, ReLU, 2 × 2 Average Pool, 1 × 1 Conv 24 × 7 ×
Dense Block(cid:20) BN, 1 × 1 Conv
BN, 3 × 3 Conv (cid:21) ×48 × 7 ×
Pooling
BN, ReLU, 7 × 7 Average Pool, Reshape
Fully connected
Linear d
B A PPENDIX : P ROOFS
In this appendix, we prove Lemmas 2.1 and 4.1, and Theorems 4.2 and 4.3.
B. P ROOF OF L EMMA 2.
Proof.
By assumption µ and γ d are both absolutely continuous with respect to the Lebesgue measure.
The desired result holds since it is a spacial case of the well known results on the existence of optimal transport (Brenier, 1991; McCann, 1995), see, Theorem 1.28 on page 24 of (Philippis, 2013) for details.
[Table] [Table]
B. P ROOF OF L EMMA 4.
Proof.
Our proof follows Keziou (2003).
Since f ( t ) is convex, then ∀ t ∈ R , we have f ( t ) =
f ∗∗ ( t ) , where
[Equation]
is the Fenchel conjugate of f ∗ .
By Fermat’s rule, the maximizer s ∗ satisﬁes
[Equation]
i.e.,
[Equation]
Plugging the above display with t = d µ Z d γ ( x ) into the deﬁnition of f -divergence, we derive (6).
B. P ROOF OF T HEOREM 4.
Proof.
Without loss of generality, we assume d = 1 .
For R ∗ satisfying (3) and any R ∈ R , we have R =
ρ
( R,R ∗ ) R ∗ + ε R , where ρ ( R,R ∗ ) is the correlation coefﬁcient between R and R ∗ , ε R =
R − ρ ( R,R ∗ ) R ∗ .
It is easy to see that ε R
R ∗ and thus Y ε R .
As ( ρ ( R,R ∗ ) R ∗ , Y ) is independent of ( ε R , 0) , then by Theorem 3 of Sz´ekely & Rizzo (2009)
[Equation]
As R ( x ) ∼ N (0 , 1) and R ∗ ( x ) ∼ N (0 , 1) , then D f ( µ R ( x ) (cid:107) γ d ) =
D f ( µ R ∗ ( x ) (cid:107) γ d ) = 0 , and
L ( R ) − L ( R ∗ ) =
V [ R ∗ , y ] − V
[ R, y ] ≥ 0 .
The proof is completed.
B. P ROOF OF T HEOREM 4.
Denote B 2 = max {| f (cid:48)
( c 1 ) | , | f (cid:48) ( c 2 ) |} , B 3 = max | s |≤ 2 L 2 √ d log n + B 2 | f ∗ ( s ) | .
We set the network parameters of the representer R θ and the discriminator D φ as follows.
(i) Representer network R D , W , S , B parameters: depth
[Equation]
9 log n
[Equation] [Equation]
width
[Equation]
(ii) Discriminator network
[Equation]
˜ D
[Equation]
˜ W , ˜ S , ˜ B parameters: depth ˜ D
= 9 log n + 12 , width ˜
[Equation]
Before getting into the details of the proof of Theorem 4.3, we ﬁrst give an outline of the basic structure of the proof.
Without loss of generality, we assume that λ
= 1 and m = 1 , i.e. y ∈ R .
First we consider the scenario that y is bounded almost surely, say | y | ≤ C 1 .
We also assume B 1 < ∞ .
We can utilize the truncation technique to transfer the unbounded cases to the bounded ones under some common tail assumptions.
Consequently, an additional log n multiplicative term will appear in the ﬁnal results.
For any ¯ R ∈ N D , W , S , B , we have,
[Equation]
where we use the deﬁnition of (cid:98) R θ in (8) and the feasibility of ¯ R .
Next we bound the two error terms in (10), i.e., the approximation error inf ¯ R ∈N D , W , S , B |L ( ¯ R ) − L ( R ∗ ) | and the statistical error sup R ∈N D , W , S , B |L ( R ) − (cid:98) L ( R ) | separately.
Then Theorem 4.3 follows after bounding these two error terms.
B.4. T HE APPROXIMATION ERROR
[Equation]
B.1.
[Equation]
Proof.
By (3) and (6) and the deﬁnition of L , we have
[Equation]
where ¯ R ¯ θ ∈ N D , W , S , B is speciﬁed in Lemma B.2 below.
We ﬁnish the proof by (14) in Lemma B. and (15) in Lemma B.4, which will be proved below.
Lemma B.2.
Deﬁne ˜ R ∗ ( x ) =
min { R ∗ ( x ) , log n } .
There exist a ¯ R ¯ θ ∈ N D , W , S , B with depth
D = 9 log n + 12 , width W =
d max { 8 d ( n p 2+ p / log n ) 1 p + 4 d, 12 n p 2+ p / log n + 14 } , and size
S = dn p
[Equation]
2 p +2 / (log 4 npd ) , B = (2 B 1 L 1 √ p + log n ) √ d, such that
[Equation]
Proof.
Let ˜ R ∗
i ( x ) be the i -th entry of ˜ R ∗ ( x ) : R d → R d .
By the assumption on R ∗ , it is easy to check that ˜ R ∗ i ( x ) is Lipschitz continuous on [ − B 1 , B 1 ] d with the Lipschitz constant L 1 and (cid:107)
˜ R ∗ i (cid:107) L ∞ ≤ log n .
By Theorem 4.3 in Shen et al.
(2019), there exists a ReLU network ¯ R ¯ θ i with with depth 9 log n + 12 , width max { 8 d ( n p 2+ p / log n ) 1 p + 4 d, 12 n p 2+ p / log n + 14 } ,
[Equation]
¯ R ¯ θ i
[Equation]
2 B 1 L 1 √ p + log n, such that
[Equation]
and
[Equation]
Deﬁne ¯ R ¯ θ =
[ ¯ R ¯ θ 1 , . .
, ¯ R ¯ θ d ] ∈ N D , W , S , B .
The above three display implies
[Equation]
Lemma B.3.
[Equation]
Proof.
Recall that Sz´ekely et al.
[Equation]
where ( z i , y i ) , i = 1 , 2 , 3 are i.i.d.
copies of ( z , y ) .
We have |V
[ R ∗ ( x ) , y ] − V
[ ¯ R ¯ θ ( x ) , y ] | ≤ | E (cid:2) ( (cid:107) R ∗ ( x 1 ) − R ∗ ( x 2 ) (cid:107) − (cid:107)
¯ R ¯ θ ( x 1 ) − ¯ R ¯ θ ( x 2 ) (cid:107) )
| y 1 − y 2 | (cid:3) | + 2 | E (cid:2) ( (cid:107) R ∗ ( x 1 ) − R ∗ ( x 2 ) (cid:107) − (cid:107)
¯ R ¯ θ ( x 1 ) − ¯ R ¯ θ ( x 2 ) (cid:107) )
|
[Equation] [Equation]
| (cid:3) |
[Equation] [Equation]
≤ 16 C
[Equation]
≤ 16 C
[Equation]
[Equation] [Equation]
) −
[Equation] [Equation]
(cid:107) (cid:105)
[Equation]
(cid:2)
[Equation] [Equation]
[Equation] [Equation]
Ball
[Equation]
( 0 , log
[Equation]
(cid:107) (cid:3)
[Equation]
where in the ﬁrst and third inequalities we use the triangle inequality, and second one follows from the boundedness of y .
By (13), the ﬁrst term in the last line is bounded by 2560 C 1 B 1 L 1 √ pdn − 1 p +2 .
Some direct calculation shows that
[Equation]
We ﬁnish the proof by comparing the order of the above two terms, i.e., C 2 (log
[Equation]
n
[Equation]
Lemma B.4.
[Equation]
Proof.
By Lemma B.2 ¯ R ¯ θ can approximate R ∗ arbitrary well, the desired result follows from the fact that D f ( µ R ∗ ( x ) (cid:107) γ d )
=
and the continuity of D f ( µ R ( x ) (cid:107) γ d ) on R .
We present the sketch of the proof and omit the details here.
Let r ∗ ( z ) =
d µ R ∗ ( x ) d γ d ( z ) and ¯ r ( z ) =
d µ ¯ R ¯ θ ( x ) d γ d ( z ) .
By deﬁnition we have
[Equation]
(We can represent D f ( µ ¯ R ¯ θ (cid:107) γ d ) similarly. )
Then
[Equation]
The ﬁrst term in the above display is small due to ¯ R ¯ θ can approximate R ∗ well.
The second term is small due to the boundedness of ¯ r and the exponential decay of the Gaussian tails.
B.4. T HE STATISTICAL ERROR
Lemma B.5.
[Equation]
Proof.
By the deﬁnition and the triangle inequality we have
[Equation]
We ﬁnish the proof based on (17) in Lemma B.6 and (22) in Lemma B.
[Equation]
which will be proved below.
Lemma B
[Equation] [Equation]
Proof.
We ﬁrst ﬁx some notation for simplicity.
Denote O = ( x , y ) ∈
R p × R 1 and O i =
( x i , y i ) , i = 1 , ...n are i.i.d copy of O , and denote µ x , y and P (cid:78) n as P and P n , respectively.
∀ R ∈ N D , W , S , B , let ˜ O
=
( R ( x ) , y ) and ˜
O i =
( R ( x i ) , y i ) , i = 1 , ...n are i.i.d copy of ˜ O .
Deﬁne centered kernel
[Equation]
Then, the centered U -statistics (cid:98) V n [ R ( x ) , y ] − V
[ R ( x ) , y ] can be represented as
[Equation]
Our goal is to bound the supremum of the centered U -process U n (¯ h R ) with the nondegenerate kernel ¯ h R .
By the symmetrization randomization Theorem 3.5.3 in De la Pena & Gin ´ e (2012), we have
E [
[Equation]
where, (cid:15) i 1 , i 1 = 1 , ...n are i.i.d Rademacher variables that are also independent with ˜
O i , i = 1 , . .
, n. We ﬁnish the proof by upper bounding the above Rademacher process with the matric entropy of N D , W , S , B .
To this end we need the following lemma.
Lemma B.7.
If ξ i , i = 1 , ...m are m ﬁnite linear combinations of Rademacher variables (cid:15) j , j = 1 , ..J .
Then
[Equation]
Proof.
This result follows directly from Corollary 3.2.6 and inequality (4.3.1) in De la Pena & Gin ´ e (2012) with Φ( x ) = exp( x 2 ) .
By the boundedness assumption on y and the boundedness of R ∈ N D , W , S , B , we have that the kernel ¯ h R is also bounded, say
[Equation]
[Equation]
∀ R, ˜ R ∈ N D , W , S , B deﬁne a random empirical measure (depends on O i , i = 1 , . . . , n )
[Equation]
Condition on O i , i = 1 , . . . , n , let C ( N , e n, 1 , δ )) be the covering number of N D , W , S , B with respect to the empirical distance e n, 1 at scale of δ > 0 .
Denote N δ as the covering set of N D , W , S , B with cardinality of C ( N , e n, 1 , δ )) .
Then,
[Equation]
where the ﬁrst inequality follows from the triangle inequality
[Equation]
the second inequality uses (20), the third and fourth inequalities follow after some algebra, and the ﬁfth inequality holds due to C ( N , e n, 1 , δ ) ≤ C ( N , e n, ∞ , δ ) and the relationship between the metric entropy and the VCdimension of the ReLU networks N D , W , S , B (Anthony & Bartlett, 2009), i.e.,
[Equation]
and the last inequality holds due to the upper bound of VC-dimension for the ReLU network
N D , W , S , B satisfying
[Equation]
see Bartlett et al.
Then (17) holds by the selection of the network parameters and set δ = 1 n and some algebra.
Lemma B.8.
[Equation]
Proof.
∀ R ∈ N D , W , S , B , let r ( z ) =
d µ R ( x ) d γ d ( z ) , g R ( z ) =
f (cid:48) ( r ( z )) .
By assumption g R ( z ) :
R d → R is Lipschitz continuous with the Lipschitz constant L 2 and (cid:107) g R (cid:107) L ∞ ≤ B 2 .
Without loss of generality, we assume supp( g R ) ⊆
[ − log n, log n ] d .
Then, similar to the proof of
Lemma B.2 we can show that there exists a ¯ D ¯ φ ∈ M ˜ D , ˜ W , ˜ S , ˜ B with depth
˜ D
= 9 log n + 12 , width ˜ W = max { 8 d ( n d 2+ d / log n )
d
+ 4 d, 12 n d 2+ d / log n + 14 } , and size ˜ S
= n d − 2 d +2 / (log 4 npd ) , ˜ B
= 2 L 2 √ d log n + B 2 such that for z ∼ γ d and z ∼ µ R ( x )
[Equation]
∀ g : R d → R , deﬁne
[Equation]
By (6) we have
[Equation]
Then,
[Equation]
where we use the triangle inequality in the ﬁrst inequality
[Equation]
and we use E ( g R ) ≥ sup D φ ∈M ˜ D , ˜ W , ˜ S , ˜ B E ( D φ ) followed from (24) and the triangle inequality in the second inequality
[Equation]
the third inequality follows from the triangle inequality
[Equation]
and the last inequality follows from (23) and the mean value theorem.
We ﬁnish the proof via bounding the empirical process
[Equation]
Let S =
( x , z ) ∼ µ x (cid:78) γ d and S i , i = 1 , . .
. , n be n i.i.d copy of S .
Denote b ( D, R ; S ) =
D ( R ( x )) − f ∗ ( D ( z )) .
Then
E ( D, R ) =
[Equation] [Equation]
( D, R ; S )] and
[Equation]
i ) .
Let
[Equation]
be the Rademacher complexity of M ˜ D , ˜ W , ˜ S , ˜ B × N D , W , S , B (Bartlett & Mendelson, 2002).
Let
C ( M × N , e n, 1 , δ )) be the covering number of M ˜ D , ˜ W , ˜ S , ˜ B × N D , W , S , B with respect to the empirical distance (depends on S i )
[Equation]
) | ] at scale of δ > 0 .
Let M δ ×
N δ be such a converging set of M ˜ D , ˜ W , ˜ S , ˜ B × N D , W , S , B .
Then,
[Equation]
i =1 ,...,n [ G ( N × M ) | ( S 1 , ..., S n )]]
[Equation] [Equation]
[Equation] [Equation]
n [ sup
[Equation]
R )
[Equation]
| n
[Equation]
(cid:15)
[Equation] [Equation]
[Equation]
|
[Equation] [Equation]
n )]
[Equation] [Equation]
[Equation] [Equation]
N
[Equation]
e n
[Equation]
δ ))
[Equation]
/max
[Table]
[Equation] [Equation]
[Equation]
[ n
[Equation]
i
[Equation]
[Table]
[Equation] [Equation]
[Equation] [Equation]
[Equation] [Equation]
where the ﬁrst equality follows from the standard symmetrization technique, the second equality holds due to the iteration law of conditional expectation, the ﬁrst inequality follows from the triangle inequality, and the second inequality uses equation 20, the third inequality uses the fact that b ( D, R ; S ) is bounded, i.e., (cid:107) b ( D, R ; S ) (cid:107) L ∞ ≤ 2 L 2 √ d log n + B 2 + B 3 , and the fourth inequality follows from some algebra, and the ﬁfth inequality follows from C ( N , e n, 1 , δ ) ≤ C ( N , e n, ∞ , δ ) (similar result for M ) and log C ( N , e n, ∞ , δ ))
≤ VC N log 2 e B n δ VC N , and N D , W , S , B satisfying
C 8 DS log S ≤ VC N ≤ C 9 DS log S , see Bartlett et al.
Then (22) follows from the above display with the selection of the network parameters of M ˜ D , ˜ W , ˜ S , ˜ B , N D , W , S , B and with δ = 1 n .
Finally, Theorem 4.3 is a direct consequence of (11) in Lemma B.1 and (16) in Lemma B.5.
This completes the proof of Theorem 4.3.
(cid:3)