S UFFICIENT
AND
D ISENTANGLED
R EPRESENTATION
L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
We propose a novel representation learning approach called sufﬁcient and disentangled representation learning (SDRL).
With SDRL, we seek a data representation that maps the input data to a lower-dimensional space with two properties: sufﬁciency and disentanglement.
First, the representation is sufﬁcient in the sense that the original input data is conditionally independent of the response or label given the representation.
Second, the representation is maximally disentangled with mutually independent components and is rotation invariant in distribution.
We show that such a representation always exists under mild conditions on the input data distribution based on optimal transport theory.
We formulate an objective function characterizing conditional independence and disentanglement.
This objective function is then used to train a sufﬁcient and disentangled representation with deep neural networks.
We provide strong statistical guarantees for the learned representation by establishing an upper bound on the excess error of the objective function and show that it reaches the nonparametric minimax rate under mild conditions.
We also validate the proposed method via numerical experiments and real data analysis.
I NTRODUCTION
Representation learning is a fundamental problem in machine learning and artiﬁcial intelligence (Bengio et al., 2013).
Certain deep neural networks are capable of learning effective data representation automatically and achieve impressive prediction results.
For example, convolutional neural networks, which can encode the basic characteristics of visual observations directly into the network architecture, is able to learn effective representations of image data (LeCun et al., 1989).
Such representations in turn can be subsequently used for constructing classiﬁers with outstanding performance.
Convolutional neural networks learn data representation with a simple structure that captures the essential information through the convolution operator.
However, in other application domains, optimizing the standard cross-entropy and least squares loss functions do not guarantee that the learned representations enjoy any desired properties (Alain & Bengio, 2016).
Therefore, it is imperative to develop general principles and approaches for constructing effective representations for supervised learning.
There is a growing literature on representation learning in the context deep neural network modeling.
Several authors studied the internal mechanism of supervised deep learning from the perspective of information theory (Tishby & Zaslavsky, 2015; Shwartz-Ziv & Tishby, 2017; Saxe et al., 2019), where they showed that training a deep neural network that optimizes the information bottleneck (Tishby et al., 2000) is a trade-off between the representation and prediction at each layer.
To make the information bottleneck idea more practical, deep variational approximation of information bottleneck (VIB) is considered in Alemi et al.
Information theoretic objectives describing conditional independence such as mutual information are utilized as loss functions to train a representation-learning function, i.e., an encoder in the unsupervised setting (Hjelm et al., 2018;
Oord et al., 2018; Tschannen et al., 2019; Locatello et al., 2019; Srinivas et al., 2020).
There are several interesting extensions of variational autoencoder (VAE) (Kingma & Welling, 2013) in the form of VAE plus a regularizer, including beta-VAE (Higgins et al., 2017), Annealed-VAE (Burgess et al., 2018), factor-VAE (Kim & Mnih, 2018), beta-TC-VAE (Chen et al., 2018), DIP-VAE (Kumar et al., 2018). The idea of using a latent variable model has also been used in adversarial auto- encoders (AAE) (Makhzani et al., 2016) and Wasserstein auto-encoders (WAE) (Tolstikhin et al., 2018).
However, these existing works focus on the unsupervised representation learning. A challenge of supervised representation learning that distinguishes it from standard supervised learning is the difﬁculty in formulating a clear and simple objective function. In classiﬁcation, the objective is clear, which is to minimize the number of misclassiﬁcations; in regression, a least squares criterion for model ﬁtting error is usually used. In representation learning, the objective is different from the ultimate objective, which is typically learning a classiﬁer or a regression function for prediction.
How to establish a simple criterion for supervised presentation learning has remained an open question (Bengio et al., 2013).
We propose a sufﬁcient and disentangled representation learning (SDRL) approach in the context of supervised learning.
With SDRL, we seek a data representation with two characteristics: sufﬁciency and disentanglement.
In the context of representation learning, sufﬁcient means that a good representation should preserve all the information in the data about the supervised learning task.
This is a basic requirement and a long-standing principle in statistics. This is closely related to the fundamental concept of sufﬁcient statistics in parametric statistical models (Fisher, 1922). A sufﬁcient representation can be naturally characterized by the conditional independence principle, which stipulates that, given the representation, the original input data does not contain any additional information about the response variable.
In addition to the basic sufﬁciency property, the representation should have a simple statistical structure.
Disentangling is based on the general notion that some latent causes underlie data generation process: although the observed data are typically high-dimensional, complex and noisy, the underlying factors are low-dimensional, independent and have a relatively simple statistical structure.
There is a range of deﬁnitions of disentangling (Higgins et al., 2018; Eastwood & Williams, 2018; Ridgeway & Mozer, 2018; Do & Tran, 2020).
Several metrics have been proposed for the evaluation of disentangling.
However, none of these deﬁnitions and metrics have been turned into empirical criterions and algorithms for learning disentangled representations.
We adopt a simple deﬁnition of disentangling which deﬁnes a representation to be disentangled if its components are independent (Achille & Soatto, 2018). This deﬁnition requires the representation to be maximally disentangled in the sense that the total correlation is zero, where the total correlation is deﬁned as the KL divergence between the joint distribution of g ( x ) and the product of the marginal distributions of its components (Watanabe, 1960).
In the rest of the paper, we ﬁrst discuss the motivation and the theoretical framework for learning a sufﬁcient and disentangled representation map (SDRM).
This framework leads to the formulation of an objective function based on the conditional independence principle and a metric for disentanglement and invariance adopted in this work.
We estimate the target SDRM based on the sample version of the objective function using deep neural networks and develop an efﬁcient algorithm for training the SDRM.
We establish an upper error bound on the measure of conditional independence and disentanglement and show that it reaches the nonparametric minimax rate under mild regularity conditions.
This result provides strong statistical guarantees for the proposed method.
We validate the proposed SDRL via numerical experiments and real data examples.
S UFFICIENT AND DISENTANGLED REPRESENTATION
Consider a pair of random vectors (
x , y ) ∈
R p × R q , where x is a vector of input variables and y is a vector of response variables or labels.
Our goal is to ﬁnd a sufﬁcient and disentangled representation of x .
Sufﬁciency We say that a measurable map g : R p
→ R d with d ≤ p is a sufﬁcient representation of x if
[Equation]
that is, y and x are conditionally independent given g ( x ) .
This condition holds if and only if the conditional distribution of y given x and that of y given g ( x ) are equal.
Therefore, the information in x about y is completely encoded by g ( x ) .
Such a g always exists, since if we simply take g ( x ) = x , then (1) holds trivially.
This formulation is a nonparametric generalization of the basic condition in sufﬁcient dimension reduction (Li, 1991; Cook, 1998), where it is assumed g ( x ) =
B T x with
B ∈ R p ×
d belonging to the Stiefel manifold, i.e., B T B = I d .
Denote the class of sufﬁcient representations satisfying (1) by
[Equation]
We refer to F as a Fisher class because of its close connection with the concept of sufﬁcient statistics (Fisher, 1922; Cook, 2007).
For an injective measurable transformation T : R d → R d and g ∈ F ,
T ◦ g ( x ) is also sufﬁcient by the basic property of conditional probability.
Therefore, the Fisher class F is invariant in the sense that
T ◦ F = F , provided T is injective, where T ◦ F = { T ◦ g : g ∈ F} .
An important class of transformations is the class of afﬁne transformations, T ◦ g =
Ag + b , where A is a d × d nonsingular matrix and b ∈ R d .
Disentanglement We focus on the disentangled representations among those that are sufﬁcient.
Therefore, we start from the functions of the input data that are sufﬁcient representations in the Fisher class F .
For any sufﬁcient and disentangled representation g ( x ) , let Σ g =
Var( g ( x )) .
Since the components of g ( x ) are disentangled in the sense that they are independent, Σ g is a diagonal matrix, thus Σ − 1 / 2 g g ( x ) also has independent components.
Therefore, we can always rescale g ( x ) such that it has identity covariance matrix.
To further simplify the statistical structure of a representation g , we also require it to be rotation invariant in distribution, that is, Qg ( x ) =
g ( x ) in distribution for any orthogonal matrix Q ∈ R d × d .
The Fisher class F is rotation invariant in terms of conditional independence, but not all its members are rotation invariant in distribution.
By the Maxwell characterization of the Gaussian distributions (Maxwell, 1860; Bartlett, 1934; Bryc, 1995; Gyenis, 2017), a random vector of dimension two or more with independent components is rotation invariant in distribution if and only if it is Gaussian with zero mean and a spherical covariance matrix.
Therefore, after absorbing the scaling factor, for a sufﬁcient representation map to be disentangled and rotation invariant, it is necessarily distributed as N d ( 0 , I d ) .
Let M be the Maxwell class of functions
g : R d → R d , where g ( x ) is disentangled and rotation invariant in distribution.
By the
Maxwell characterization, we can write
[Equation]
Now our problem becomes that of ﬁnding a representation in F ∩ M , the intersection of the Fisher class and the Maxwell class.
The ﬁrst question to ask is whether such a representation exists.
The following result from optimal transport theory provides an afﬁrmative answer and guarantees that F ∩ M is nonempty under mild conditions (Brenier, 1991; McCann, 1995; Villani, 2008).
Lemma 2.1.
Let µ be a probability measure on R d .
Suppose it has ﬁnite second moment and is absolutely continuous with respect to the standard Gaussian measure, denoted by γ d .
Then it admits a unique optimal transportation map T : R d → R d such that T # µ = γ d ≡ N ( 0 , I d ) , where T # µ denotes the pushforward distribution of µ under T .
Moreover, T is injective µ -almost everywhere.
Denote the law of a random vector z by µ z .
Lemma 2.1 implies that, for any g ∈ F with E (cid:107) g
( x ) (cid:107) 2 < ∞ and µ g ( x ) absolutely continuous with respect to γ d , there exists a map T ∗ transforming the distribution of g ( x ) to N ( 0 , I d ) .
Therefore, R ∗ :=
T ∗ ◦ g ∈ F ∩ M , that is, [Equation] ∗ [Equation]
x
[Equation]
and R ∗ ( x ) ∼ N
[Equation]
I d ) , ( [Equation] ) i.e., R ∗ is a sufﬁcient and disentangled representation map (SDRM).
O BJECTIVE FUNCTION FOR SDRL
The above discussions lay the theoretical foundation for formulating an objective function that can be used for constructing a SDRM R ∗ satisfying (3), or equivalently, R ∗ ∈ F ∩ M .
Let V be a measure of dependence between random variables x and y with the following properties: (a
) V [ x , y ] ≥ 0 with V [ x , y
] =[-
-] if and only if x y ; ( b ) V [ x y ] ≥ V
[ R ( x ) , y
] for all measurable function R ; (c) V [ x , y ]
=
V [ R ∗ ( x ) , y ] if and only if R ∗ ∈ F .
The properties (a)-(c) imply that
[Equation]
We use a divergence measure D to quantify the difference between µ R ( x ) and γ d , as long as this measure satisﬁes the condition D ( µ R ( x ) (cid:107) γ d ) ≥ 0 for all measurable function R and D ( µ R ( x ) (cid:107) γ d ) =
if and only if R ∈ M .
Then the problem of ﬁnding an R ∗ ∈ F ∩ M can be expressed as a constrained minimization problem:
[Equation]
] subject to
[Equation]
Its Lagrangian form is
[Equation]
where λ ≥ 0 is a tuning parameter.
This parameter provides a balance between the sufﬁciency property and the disentanglement constraint.
A small λ leads to a representation with more emphasis on sufﬁciency, while a large λ yields a representation with more emphasis on disentanglement.
We show in Lemma 4.1 below that any R ∗ satisfying (3) is a minimizer of L ( R ) .
Therefore, we can train a SDRM by minimizing the empirical version of L ( R ) .
There are several options for V with the properties (a)-(c) described above.
For example, we can take
V to be the mutual information V
[ R ( x ) , y ]
= I ( R ( x ); y ) .
However, in addition to the estimation of the SDRM R , this choice requires the estimation of the density ratio between p ( y , R ( x )) and p ( y ) p ( R ( x )) , which is not an easy task.
We can also use the conditional covariance operators on reproducing kernel Hilbert spaces (Fukumizu et al., 2009).
To be speciﬁc, in this work we use the distance covariance (Sz ´ ekely et al., 2007) of y and R ( x ) , which has an elegant U -statistic expression, does not involve additional unknown quantities and is easy to compute.
For the divergnce measure of two distributions, we use the f -divergence (Ali & Silvey, 1966), which includes the KLdivergence as a special case.
L EARNING SUFFICIENT AND DISENTANGLED REPRESENTATION
We ﬁrst describe some essentials about distance covariance and f -divergence.
Distance covariance We ﬁrst recall the concept of distance covariance (Sz ´ ekely et al., 2007), which characterizes the dependence of two random variables.
Let i be the imaginary unit ( − 1) 1 / 2 .
For any t ∈ R d and s ∈ R m , let ψ z ( t ) =
E [exp i t T z ] , ψ y ( s ) =
E [exp i s T y ] , and ψ z , y ( t , s ) =
E [exp i ( t T z + s T y ) ] be the characteristic functions of random vectors z ∈ R d , y ∈ R q , and the pair ( z , y ) , respectively.
The squared distance covariance V [ z , y ] is deﬁned as
[Equation]
where c d = π
[Equation]
d +1) /Γ [Equation] [Equation] Given n i.i.d copies { z i , y i } ni =1 of ( z , y ) , an unbiased estimator of V is the empirical distance covariance (cid:98) V n , which can be elegantly expressed as a U -statistic
(Huo & Sz ´ ekely, 2016)
[Equation]
where h is the kernel deﬁned by
[Equation]
[Equation] [Equation] f-divergence Let µ and γ be two probability measures on R d .
The f -divergence (Ali & Silvey, 1966) between µ and γ with µ (cid:28) γ is deﬁned as D f ( µ (cid:107) γ ) =
(cid:82) R d f ( d µ d γ )d γ, where f : R + → R is a differentiable convex function satisfying f (1) = 0 .
Let f ∗ be the Fenchel conjugate of f (Rockafellar, 1970), deﬁned as f ∗ ( t ) =
sup x ∈ R { tx − f ( x ) } , t ∈ R .
The f -divergence admits the following variational formulation (Keziou, 2003;
Nguyen et al., 2010; Nowozin et al., 2016).
Lemma 4.1.
[Equation]
where the maximum is attained at D ( z )
=
f (cid:48)
Commonly used divergence measures include the Kullback-Leibler (KL) divergence, the JensenShanon (JS) divergence and the χ 2 -divergence.
Learning SDRM We are now ready to formulate an empirical objective function for learning SDRM R ∗ .
Let R ∈ M , where M is the Maxwell class deﬁned in (2).
By the variational formulation (6), we can write the population version of the objective function (4) as
[Equation]
This expression is convenient since we can simply replace the expectations by the corresponding empirical averages.
Theorem 4.2.
We have R ∗ ∈ arg min R ∈M
L ( R ) provided (3) holds.
According to Theorem 4.2, it is natural to estimate R ∗ based on the empirical version of the objective function (7) when a random sample { ( x i , y i ) } ni =1 is available.
We estimate R ∗ using deep neural networks.
We employ two networks as follows:
• Representer network R θ : This network is used for training R ∗ .
Let R be the set of such neural networks R θ : R p
→ R d . • Discriminator network D φ : This network is used as the witness function for checking whether the distribution of the estimator of R ∗ is approximately the same as N ( 0 , I d ) .
Similarly, denote D as the set of such neural networks D φ : R d → R
Let { w i } ni =1 be n i.i.d random vectors drawn from γ d .
The estimated SDRM is deﬁned by
[Equation]
where (cid:98) V n [ R θ ( x ) , y ] is an unbiased and consistent estimator of V [ R θ ( x ) , y ] as deﬁned in (5) based on { ( R θ ( x i ) , y i ) , i = 1 , . .
. , n } and
[Equation]
Statistical guarantee Since a SDRM R ∗ is only identiﬁable up to orthogonal transforms under the constraint that R ∗ ( x ) ∼ N (0 , I d ) , no consistency results for (cid:98) R θ itself can be obtained.
But this is not a ﬂaw of the proposed method.
Indeed, the most important statistical guarantee of the learned
R ∗ is that the objective of conditional independence and disentanglement is achieved.
Therefore, we establish an upper bound on the excess risk L ( (cid:98) R θ )
−L ( R ∗ ) of the deep nonparametric estimator (cid:98) R θ in (8).
We make the following assumptions.
(A1)
For any ε > 0 , there is a constant B 1 > 0 such that µ x ([ − B 1 , B 1 ] p ) > 1 − ε , and R ∗ is Lipschitz continuous on [ − B 1 , B 1 ] p with Lipschitz constant L 1 .
(A2)
For R ∈ M , we assume r ( z ) =
d µ R ( x ) d γ d ( z ) is Lipschitz continuous on [ − B 1 , B 1 ] p with Lipschitz constant L 2 , and 0 < c 1 ≤ r ( z ) ≤ c 2 .
Denote
[Equation]
The speciﬁcations of the network parameters, including depth, width, size and the supremum norm over the domains of the representer R θ and the discriminator D φ are given in Appendix B.
Theorem 4.3.
Suppose λ > 0 and λ = O (1) .
Suppose conditions (A1)-(A2) hold and set the network parameters according to (i)-(ii).
Then
[Equation]
where C is a constant that depends on B 1 , B 2 and B 3 but not on n, q, p and d .
The proof of this theorem is given in Appendix B.
The result established in Theorem 4.3 provides strong statistical guarantees for the proposed method.
The rate n − 2 / (2+ p ) matches the minimax nonparametric estimation rate for Lipschitz class contained in R p of functions (Stone, 1982; Tsybakov, 2008).
Up to a log n factor, the rate (log n ) n − 2 / (2+ d ) matches the minimax rate of nonparametric estimation of Lipschitz densities via GANs (Singh et al., 2018; Liang, 2018).
C OMPUTATION
We can update θ and φ alternately as in training GANs (Goodfellow et al., 2014).
However, this approach suffers from the instability issues.
In our implementation, we utilize the more stable particle method based on gradient ﬂow (Gao et al., 2019; 2020).
The key idea is to ﬁnd a sequence of nonlinear but simpler residual maps, say T ( z )
=
z + s v ( z ) , pushing the samples from µ R θ ( x ) to the target distribution γ d along a velocity ﬁelds v ( z ) =
−∇ f (cid:48)
( r ( z )) that most decreases the f -divergence D f ( ·|| γ d ) at µ R θ ( x ) .
The residual maps can be estimated via deep density-ratio estimators, which take the form T ( z ) =
z + s (cid:98) v ( z ) , z ∈ R d , where s is a step size and (cid:98) v ( z ) =
− f (cid:48)(cid:48)
(ˆ r ( z )) ∇ ˆ r ( z ) .
Here ˆ r ( z ) is an estimated density ratio of the density of R θ ( x ) at the current value of θ over the density of the reference distribution.
We use T to transform z i =
R θ ( x i ) , i = 1 , . . . , n into Gaussian samples.
Once this is done, we update θ via minimizing the loss − (cid:98) V n [ R θ ( x ) , y ]+ λ (cid:80) ni
=1 (cid:107)
R θ ( x i ) −
z i (cid:107) 2 /n.
We describe the algorithm below.
• Input { x i , y i } ni =1 .
Tuning parameters: s, λ, d .
Sample { w i } ni
=1 ∼ γ d . • Outer loop for θ – Inner loop (particle method) ∗ Let z i
= R θ ( x i ) , i = 1 , 2 , ..., n .
∗ Solve (cid:98) D φ ∈ arg min Q φ (cid:80)
ni =1 1 n (cid:0) log(1 + exp D φ ( z i ) )
+ log(1 + exp − D φ ( w i ) ) (cid:1) .
∗ Deﬁne the residual map T ( z ) =
z − sf (cid:48)(cid:48)
(ˆ r ( z )) ∇ ˆ r ( z ) with ˆ r ( z ) =
exp − (cid:98) D
φ ( z ) .
∗ Update the particles z i
= T ( z i ) , i = 1 , 2 , ..., n .
– End inner loop – Update θ via minimizing − (cid:98) V n [ R θ ( x ) , y ] + λ (cid:80) ni
=1 (cid:107)
R θ ( x i ) −
z i (cid:107) 2 /n using SGD.
• End outer loop
E XPERIMENTS
We evaluate the proposed SDRL with the KL-divergence using both simulated and real data.
The goal of our experiments is to demonstrate that the representations trained based the proposed method perform well.
Our proposed method is not trying to learn a classiﬁer or a regression function directly, but rather to learn representation that preserve all the information.
So our experiments are designed to evaluate the performance of simple classiﬁcation and regression methods using the representations we learned as input.
The results demonstrate that a simple classiﬁcation or regression model using the representations we trained performs better than or comparably with the best classiﬁcation or regression method using deep neural networks.
Details on the network structures and hyperparameters are included in
Appendix A.
Our experiments were conducted on Nvidia DGX Station workstation using a single Tesla V100 GPU unit.
The PyTorch code of SDRL is available at https://github.com/anonymous/SDRL .
6. S IMULATED DATA
In this subsection, we evaluate SDRL on simulated regression and classiﬁcation problems.
Regression
We generate 5 , 000 data points from two models.
Model A: y = [Equation]
[Equation] ε , where x ∼ N ( 0 , I 4 ) ; Model B: y = sin 2 ( π x 1 + 1) + σ ε , where x ∼ Uniform([0 , 1] 4 ) .
In both models, ε ∼ N ( 0 , I 4 ) .
We use a 3-layer network with
ReLU activation for R θ and a single hidden layer ReLU network for D φ .
We compare SDRL with two prominent sufﬁcient dimension reduction methods: sliced inverse regression (SIR) (Li, 1991) and sliced average variance estimation (SAVE) (Cook & Weisberg, 1991).
We ﬁt a linear model with the learned features and the response variable, and report the prediction errors in Table 1.
We see that SDRL outperforms SIR and SAVE in terms of prediction error.
Classiﬁcation We visualize the learned features of SDRL on two simulated datasets.
We ﬁrst generate (1) 2-dimensional concentric circles from two classes as in Figure 1 (a); (2) 2-dimensional moons data from two classes as in Figure 1 (e); (3) 3-dimensional Gaussian data from six classes [Table] as in Figure 1 (i).
In each dataset, we generate 5,000 data points for each class.
We next map the data into 100-dimensional space using matrices with entries i.i.d Unifrom([0 , 1]) .
Finally, we apply SDRL to these 100 -dimensional datasets to learn 2 -dimensional features.
We use a 10-layer dense convolutional network (DenseNet) (Huang et al., 2017) as R θ and a 4-layer network as D φ .
We display the evolutions of the learned 2-dimensional features by SDRL in Figure 1.
For ease of visualization, we push all the distributions onto the uniform distribution on the unit circle, which is done by normalizing the standard Gaussian random vectors to length one.
Clearly, the learned features for different classes in the examples are well disentangled.
[Equation]
, second and third rows show concentric circles, moons and 3D
, respectively.
6. R EAL DATASETS
Regression We use a benchmark YearPredictionMSD dataset to demonstrate the prediction performance of SDRL ( https://archive.ics.uci.edu/ml/datasets/YearPredictionMSD ) .
This dataset has 515,345 observations with 90 predictors.
The problem is to predict the year of song release.
We randomly split the data into ﬁve parts for cross validated evaluation of the prediction performance.
We employ a 3-layer network for both D φ and R θ .
A linear regression model is ﬁtted using the learned representations and the response.
The mean prediction errors and their standard errors based on SDRL, principal component analysis (PCA), sparse principal component analysis (SPCA) and ordinary least squares (OLS) regression with original data are reported in Table 2.
SDRL outperforms PCA, SPCA and OLS in terms prediction accuracy.
Classiﬁcation
We benchmark the classiﬁcation performance of SDRL using MNIST (LeCun et al., 2010), FashionMNIST (Xiao et al., 2017), and CIFAR-10 (Krizhevsky et al., 2009) against alterna- [Table] tive methods including convolutional networks (CN) and distance correlation autoencoder (dCorAE) (Wang et al., 2018).
With CN, we use the feature extractor by dropping the cross entropy layer of the DenseNet trained for classiﬁcation.
The MNIST and FashionMNIST datasets consist of 60k and 10k grayscale images with 28 × 28 pixels for training and testing, respectively, while the CIFARdataset contains 50k and 10k colored images with 32 × 32 pixels for training and testing, respectively.
For the learning from scratch strategy, the representer network R θ has 20 layers for MNIST data and 100 layers for CIFAR-10 data.
We apply the transfer learning technique to the combination of SDRL and CN on CIFAR-10 (Krizhevsky et al., 2009).
The pretrained WideResnet-101 model (Zagoruyko & Komodakis, 2016) on the Imagenet dataset with Spinal FC (Kabir et al., 2020) is adopt for R θ .
The discriminator network D φ is a 4-layer network.
The the architecture of R θ
and most hyperparameters are shared across all four methods - SDRL, CN, SDRL+CN and dCorAE.
Finally, we use the k -nearest neighbor ( k = 5 ) classiﬁer on the learned features for all methods.
The classiﬁcation accuracies are reported in Tables 3 and 4.
We can see that the classiﬁcation accuracies of SDRL are comparable with those of CN and dCorAE.
As shown in Table 4, the classiﬁcation accuracies of CN leveraging SDRL outperforms those of CN.
We also calculate the estimated distance correlation (DC) between the learned features and their labels as ρ 2 z , y = V [ z , y ] 2 / (cid:112) ( V [ z ] 2 × V [ y ] 2 ) , where V [ z ] and V [ y ] are the distance variances such that V [ z ] = V [ z , z ] , V [ y ]
= V [ y , y ] . For more details, please see Sz ´ ekely et al. Figureshows the DC values MNIST, FashionMNIST and CIFAR-10 data.
SDRL and SDRL+CN achieves higher DC values
[Table]
[Table]
[Table] [Table]
C ONCLUSION AND FUTURE WORK
In this work [Table] we formulate a framework for sufﬁcient and disentangled representation learning and construct an objective function characterizing conditional independence and disentanglement.
This enables us to learn a representation with the desired properties empirically.
We provide statistical guarantees for the learned representation by deriving an upper bound on the excess risk of the objective function. [Figure]
There are several questions that deserve further study.
First, we can adopt different measures of conditional independence including mutual information and conditional covariance operators on reproducing kernel Hilbert spaces (Fukumizu et al., 2009).
We can also use other divergence measures such as the Wasserstein distance in the objective function.
Finally, Lemma 2.1 suggests that the intersection of the Fisher class F and the Maxwell class M can still be large, and there can be many statistically equivalent representations in F ∩ M .
We can make further reduction of F ∩ M by imposing additional constraints, for example, certain minimal properties, sparsity, and robustness against noise perturbation.
A A PPENDIX : E XPERIMENTAL DETAILS A. S IMULATION STUDIES
The values of the hyper-parameters for the simulated experiments are given in Table A1, where λ is the penalty parameter, d is the dimension of the SDRM, n is the mini-batch size in SGD, T 1 is the number of inner loops to push forward particles z i , T 2 is the number of outer loops for training R θ , and s is the step size to update particles.
For the regression models, the neural network architectures are shown in Table A
As shown in Table A a multilayer perceptron (MLP) is utilized for the neural structure D φ in the classiﬁcation problem.
The detailed architecture of 10-layer dense convolutional network (DenseNet) (Huang et al., 2017; Amos & Kolter) deployed for R θ is shown in Table A4.
For all the settings, we adopted the Adam (Kingma & Ba, 2014) optimizer with an initial learning rate of 0.
and weight decay of 0.0001.
[Table] [Table]
[Table] [Table]
Convolution
[Table]
Dense Block(cid:20) BN, 1 × 1 Conv
BN, 3 × 3 Conv (cid:21) ×
[Table]
20 ×
Transition Layer
BN, ReLU, 2 × 2 Average Pool, 1 × 1 Conv 30 × 10 ×
Dense Block(cid:20) BN, 1 × 1 Conv
BN, 3 × 3 Conv (cid:21) ×18 × 10 ×
Transition Layer
BN, ReLU, 2 × 2 Average Pool, 1 × 1 Conv 15 × 5 ×
Dense Block(cid:20) BN, 1 × 1 Conv
BN, 3 × 3 Conv (cid:21) ×27 × 5 ×
Pooling
BN, ReLU, 5 × 5 Average Pool, Reshape
Fully connected
[Table]
A. R EAL DATASETS
Regression:
In the regression problems, hyper-parameters are presented in Table A5.
The Adam optimizer with an initial learning rate of 0.001 and weight decay of 0.0001 is adopted.
The MLP architectures of D φ and R θ for the YearPredictionMSD data are shown in Table A6. [Table] [Table]
Classiﬁcation: For the classiﬁcation problems, hyper-parameters are shown in Table
A7. We again use Adam as the SGD optimizers for both D φ and R θ .
Speciﬁcally, learning rate of 0.001 and weight decay of 0.0001 are used for D φ in all datasets and for R θ on MNIST (LeCun et al., 2010).
We customized the SGD optimizers with momentum at 0.9, weight decay at 0.0001, and learning rate ρ in Table A8 for FashionMNIST (Xiao et al., 2017) and CIFAR-10 (Krizhevsky et al., 2012). For the transfer learning of CIFAR-10, we use customized SGD optimizer with initial learning rate of 0.
and momentum of 0.9 for R θ .
MLP architectures of the discriminator network D φ for MNIST,
FashionMNIST and CIFAR-10 are given in Table A3.
The 20-layer DenseNet networks shown in
Table A9 were utlized for R θ on the MNIST dataset, while the 100-layer DenseNet networks shown in Table A
[Table]
B A PPENDIX : P ROOFS
In this appendix
[Table] we prove Lemmas 2.1 and 4. [Table] and Theorems 4.2 and 4.3.
B. P ROOF OF L EMMA 2.
Proof.
By assumption µ and γ d are both absolutely continuous with respect to the Lebesgue measure.
The desired result holds since it is a spacial case of the well known results on the existence of optimal transport (Brenier, 1991; McCann, 1995), see, Theorem 1.28 on page 24 of (Philippis, 2013) for details.
[Table]
[Table]
[Table]
[Table]
B. P ROOF OF L EMMA 4.
Proof.
Our proof follows Keziou (2003).
Since f ( t ) is convex, then ∀ t ∈ R , we have f ( t ) =
f ∗∗ ( t ) , where
[Equation]
is the Fenchel conjugate of f ∗ .
By Fermat’s rule, the maximizer s ∗ satisﬁes
[Equation]
i.e.,
[Equation]
Plugging the above display with t = d µ Z d γ ( x ) into the deﬁnition of f -divergence, we derive (6).
B. P ROOF OF T HEOREM 4.
Proof.
Without loss of generality, we assume d = 1 .
For R ∗ satisfying (3) and any R ∈ R , we have R =
ρ
( R,R ∗ ) R ∗ + ε R , where ρ ( R,R ∗ ) is the correlation coefﬁcient between R and R ∗ , ε R =
R − ρ ( R,R ∗ ) R ∗ .
It is easy to see that ε R
R ∗ and thus Y ε R .
As ( ρ ( R,R ∗ ) R ∗ , Y ) is independent of ( ε R , 0) , then by Theorem 3 of Sz´ekely & Rizzo (2009)
[Equation]
As R ( x ) ∼ N (0 , 1) and R ∗ ( x ) ∼ N (0 , 1) , then D f ( µ R ( x ) (cid:107) γ d ) =
D f ( µ R ∗ ( x ) (cid:107) γ d ) = 0 , and
L ( R ) − L ( R ∗ ) =
V [ R ∗ , y ] − V
[ R, y ] ≥ 0 .
The proof is completed.
B. P ROOF OF T HEOREM 4.
Denote B 2 = max {| f (cid:48)
( c 1 ) | , | f (cid:48) ( c 2 ) |} , B 3 = max | s |≤ 2 L 2 √ d log n + B 2 | f ∗ ( s ) | .
We set the network parameters of the representer R θ and the discriminator D φ as follows.
(i) Representer network R D , W , S , B parameters: depth
[Equation]
9 log n
[Equation] [Equation]
width
[Equation]
(ii) Discriminator network
[Equation]
˜ D
[Equation]
˜ W , ˜ S , ˜ B parameters: depth ˜ D
= 9 log n + 12 , width ˜
[Equation]
Before getting into the details of the proof of Theorem 4.3, we ﬁrst give an outline of the basic structure of the proof.
Without loss of generality, we assume that λ
= 1 and m = 1 , i.e. y ∈ R .
First we consider the scenario that y is bounded almost surely, say | y | ≤ C 1 .
We also assume B 1 < ∞ .
We can utilize the truncation technique to transfer the unbounded cases to the bounded ones under some common tail assumptions.
Consequently, an additional log n multiplicative term will appear in the ﬁnal results.
For any ¯ R ∈ N D , W , S , B , we have,
[Equation]
where we use the deﬁnition of (cid:98) R θ in (8) and the feasibility of ¯ R .
Next we bound the two error terms in (10), i.e., the approximation error inf ¯ R ∈N D , W , S , B |L ( ¯ R ) − L ( R ∗ ) | and the statistical error sup R ∈N D , W , S , B |L ( R ) − (cid:98) L ( R ) | separately.
Then Theorem 4.3 follows after bounding these two error terms.
B.4. T HE APPROXIMATION ERROR
[Equation]
B.1.
[Equation]
Proof.
By (3) and (6) and the deﬁnition of L , we have
[Equation]
where ¯ R ¯ θ ∈ N D , W , S , B is speciﬁed in Lemma B.2 below.
We ﬁnish the proof by (14) in Lemma B. and (15) in Lemma B.4, which will be proved below.
Lemma B.2.
Deﬁne ˜ R ∗ ( x ) =
min { R ∗ ( x ) , log n } .
There exist a ¯ R ¯ θ ∈ N D , W , S , B with depth
D = 9 log n + 12 , width W =
d max { 8 d ( n p 2+ p / log n ) 1 p + 4 d, 12 n p 2+ p / log n + 14 } , and size
S = dn p
[Equation]
2 p +2 / (log 4 npd ) , B = (2 B 1 L 1 √ p + log n ) √ d, such that
[Equation]
Proof.
Let ˜ R ∗
i ( x ) be the i -th entry of ˜ R ∗ ( x ) : R d → R d .
By the assumption on R ∗ , it is easy to check that ˜ R ∗ i ( x ) is Lipschitz continuous on [ − B 1 , B 1 ] d with the Lipschitz constant L 1 and (cid:107)
˜ R ∗ i (cid:107) L ∞ ≤ log n .
By Theorem 4.3 in Shen et al.
(2019), there exists a ReLU network ¯ R ¯ θ i with with depth 9 log n + 12 , width max { 8 d ( n p 2+ p / log n ) 1 p + 4 d, 12 n p 2+ p / log n + 14 } ,
[Equation]
¯ R ¯ θ i
[Equation]
2 B 1 L 1 √ p + log n, such that
[Equation]
and
[Equation]
Deﬁne ¯ R ¯ θ =
[ ¯ R ¯ θ 1 , . .
, ¯ R ¯ θ d ] ∈ N D , W , S , B .
The above three display implies
[Equation]
Lemma B.3.
[Equation]
Proof.
Recall that Sz´ekely et al.
[Equation]
where ( z i , y i ) , i = 1 , 2 , 3 are i.i.d.
copies of ( z , y ) .
We have |V
[ R ∗ ( x ) , y ] − V
[ ¯ R ¯ θ ( x ) , y ] | ≤ | E (cid:2) ( (cid:107) R ∗ ( x 1 ) − R ∗ ( x 2 ) (cid:107) − (cid:107)
¯ R ¯ θ ( x 1 ) − ¯ R ¯ θ ( x 2 ) (cid:107) )
| y 1 − y 2 | (cid:3) | + 2 | E (cid:2) ( (cid:107) R ∗ ( x 1 ) − R ∗ ( x 2 ) (cid:107) − (cid:107)
¯ R ¯ θ ( x 1 )
[Equation]
¯ R ¯ θ ( x 2 ) (cid:107)
[Equation]
|
[Equation]
| (cid:3) |
[Equation]
≤
[Equation]
≤
[Equation]
[Equation]
where in the ﬁrst and third inequalities we use the triangle inequality, and second one follows from the boundedness of y .
By (13), the ﬁrst term in the last line is bounded by 2560 C 1 B 1 L 1 √ pdn − 1 p +2 .
Some direct calculation shows that
[Equation]
We ﬁnish the proof by comparing the order of the above two terms, i.e., C 2 (log
[Equation]
n
[Equation]
Lemma B.4.
[Equation]
Proof.
By Lemma B.2 ¯ R ¯ θ can approximate R ∗ arbitrary well, the desired result follows from the fact that D f ( µ R ∗ ( x ) (cid:107) γ d )
=
and the continuity of D f ( µ R ( x ) (cid:107) γ d ) on R .
We present the sketch of the proof and omit the details here.
Let r ∗ ( z ) =
d µ R ∗ ( x ) d γ d ( z ) and ¯ r ( z ) =
d µ ¯ R ¯ θ ( x ) d γ d ( z ) .
By deﬁnition we have
[Equation]
(We can represent D f ( µ ¯ R ¯ θ (cid:107) γ d ) similarly. )
Then
[Equation]
The ﬁrst term in the above display is small due to ¯ R ¯ θ can approximate R ∗ well.
The second term is small due to the boundedness of ¯ r and the exponential decay of the Gaussian tails.
B.4. T HE STATISTICAL ERROR
Lemma B.5.
[Equation]
Proof.
By the deﬁnition and the triangle inequality we have
[Equation]
We ﬁnish the proof based on (17) in Lemma B.6 and (22) in Lemma B.
[Equation]
which will be proved below.
Lemma B
[Equation] [Equation]
Proof.
We ﬁrst ﬁx some notation for simplicity.
Denote O = ( x , y ) ∈
R p × R 1 and O i =
( x i , y i ) , i = 1 , ...n are i.i.d copy of O , and denote µ x , y and P (cid:78) n as P and P n , respectively.
∀ R ∈ N D , W , S , B , let ˜ O
=
( R ( x ) , y ) and ˜
O i =
( R ( x i ) , y i ) , i = 1 , ...n are i.i.d copy of ˜ O .
Deﬁne centered kernel
[Equation]
Then, the centered U -statistics (cid:98) V n [ R ( x ) , y ] − V
[ R ( x ) , y ] can be represented as
[Equation]
Our goal is to bound the supremum of the centered U -process U n (¯ h R ) with the nondegenerate kernel ¯ h R .
By the symmetrization randomization Theorem 3.5.3 in De la Pena & Gin ´ e (2012), we have
E [
[Equation]
where, (cid:15) i 1 , i 1 = 1 , ...n are i.i.d Rademacher variables that are also independent with ˜
O i , i = 1 , . .
, n. We ﬁnish the proof by upper bounding the above Rademacher process with the matric entropy of N D , W , S , B .
To this end we need the following lemma.
Lemma B.7.
If ξ i , i = 1 , ...m are m ﬁnite linear combinations of Rademacher variables (cid:15) j , j = 1 , ..J .
Then
[Equation]
Proof.
This result follows directly from Corollary 3.2.6 and inequality (4.3.1) in De la Pena & Gin ´ e (2012) with Φ( x ) = exp( x 2 ) .
By the boundedness assumption on y and the boundedness of R ∈ N D , W , S , B , we have that the kernel ¯ h R is also bounded, say
[Equation]
[Equation]
∀ R, ˜ R ∈ N D , W , S , B deﬁne a random empirical measure (depends on O i , i = 1 , . . . , n )
[Equation]
Condition on O i , i = 1 , . . . , n , let C ( N , e n, 1 , δ )) be the covering number of N D , W , S , B with respect to the empirical distance e n, 1 at scale of δ > 0 .
Denote N δ as the covering set of N D , W , S , B with cardinality of C ( N , e n, 1 , δ )) .
Then,
[Equation]
where the ﬁrst inequality follows from the triangle inequality
[Equation]
the second inequality uses (20), the third and fourth inequalities follow after some algebra, and the ﬁfth inequality holds due to C ( N , e n, 1 , δ ) ≤ C ( N , e n, ∞ , δ ) and the relationship between the metric entropy and the VCdimension of the ReLU networks N D , W , S , B (Anthony & Bartlett, 2009), i.e.,
[Equation]
and the last inequality holds due to the upper bound of VC-dimension for the ReLU network
N D , W , S , B satisfying
[Equation]
see Bartlett et al.
Then (17) holds by the selection of the network parameters and set δ = 1 n and some algebra.
Lemma B.8.
[Equation]
Proof.
∀ R ∈ N D , W , S , B , let r ( z ) =
d µ R ( x ) d γ d ( z ) , g R ( z ) =
f (cid:48) ( r ( z )) .
By assumption g R ( z ) :
R d → R is Lipschitz continuous with the Lipschitz constant L 2 and (cid:107) g R (cid:107) L ∞ ≤ B 2 .
Without loss of generality, we assume supp( g R ) ⊆
[ − log n, log n ] d .
Then, similar to the proof of
Lemma B.2 we can show that there exists a ¯ D ¯ φ ∈ M ˜ D , ˜ W , ˜ S , ˜ B with depth
˜ D
= 9 log n + 12 , width ˜ W = max { 8 d ( n d 2+ d / log n )
d
+ 4 d, 12 n d 2+ d / log n + 14 } , and size ˜ S
= n d − 2 d +2 / (log 4 npd ) , ˜ B
= 2 L 2 √ d log n + B 2 such that for z ∼ γ d and z ∼ µ R ( x )
[Equation]
∀ g : R d → R , deﬁne
[Equation]
By (6) we have
[Equation]
Then,
[Equation]
where we use the triangle inequality in the ﬁrst inequality
[Equation]
and we use E ( g R ) ≥ sup D φ ∈M ˜ D , ˜ W , ˜ S , ˜ B E ( D φ ) followed from (24) and the triangle inequality in the second inequality
[Equation]
the third inequality follows from the triangle inequality
[Equation]
and the last inequality follows from (23) and the mean value theorem.
We ﬁnish the proof via bounding the empirical process
[Equation]
Let S =
( x , z ) ∼ µ x (cid:78) γ d and S i , i = 1 , . .
. , n be n i.i.d copy of S .
Denote b ( D, R ; S ) =
D ( R ( x )) − f ∗ ( D ( z )) .
Then
E ( D, R ) =
[Equation] [Equation]
( D, R ; S )] and
[Equation]
i ) .
Let
[Equation]
be the Rademacher complexity of M ˜ D , ˜ W , ˜ S , ˜ B × N D , W , S , B (Bartlett & Mendelson, 2002).
Let
C ( M × N , e n, 1 , δ )) be the covering number of M ˜ D , ˜ W , ˜ S , ˜ B × N D , W , S , B with respect to the empirical distance (depends on S i )
[Equation]
) | ] at scale of δ > 0 .
Let M δ ×
N δ be such a converging set of M ˜ D , ˜ W , ˜ S , ˜ B × N D , W , S , B .
Then,
[Equation]
i =1 ,...,n [ G ( N × M ) | ( S 1 , ..., S n )]]
[Equation] [Equation]
[Equation] [Equation]
n [ sup
[Equation]
[Equation]
|
[Equation]
(cid:15)
[Equation] [Equation]
[Equation]
|
[Equation] [Equation]
n )]
[Equation]
[Equation] [Equation]
N
[Equation]
n
[Equation]
δ ))
[Equation]
[Equation]
max
[Table]
[Equation] [Equation]
[Equation]
[ n
[Equation]
i
[Equation]
[Equation]
[Equation] [Equation]
[Equation] [Equation]
[Equation] [Equation]
where the ﬁrst equality follows from the standard symmetrization technique, the second equality holds due to the iteration law of conditional expectation, the ﬁrst inequality follows from the triangle inequality, and the second inequality uses equation 20, the third inequality uses the fact that b ( D, R ; S ) is bounded, i.e., (cid:107) b ( D, R ; S ) (cid:107) L ∞ ≤ 2 L 2 √ d log n + B 2 + B 3 , and the fourth inequality follows from some algebra, and the ﬁfth inequality follows from C ( N , e n, 1 , δ ) ≤ C ( N , e n, ∞ , δ ) (similar result for M ) and log C ( N , e n, ∞ , δ ))
≤ VC N log 2 e B n δ VC N , and N D , W , S , B satisfying
C 8 DS log S ≤ VC N ≤ C 9 DS log S , see Bartlett et al.
Then (22) follows from the above display with the selection of the network parameters of M ˜ D , ˜ W , ˜ S , ˜ B , N D , W , S , B and with δ = 1 n .
Finally, Theorem 4.3 is a direct consequence of (11) in Lemma B.1 and (16) in Lemma B.5.
This completes the proof of Theorem 4.3.
(cid:3)