Nonlinear Sufﬁcient Dimension Reduction with a Stochastic Neural Network
Abstract
Sufﬁcient dimension reduction is a powerful tool to extract core information hidden in the high-dimensional data and has potentially many important applications in machine learning tasks.
However, the existing nonlinear sufﬁcient dimension reduction methods often lack the scalability necessary for dealing with large-scale data.
We propose a new type of stochastic neural network under a rigorous probabilistic framework and show that it can be used for sufﬁcient dimension reduction for large-scale data.
The proposed stochastic neural network is trained using an adaptive stochastic gradient Markov chain Monte Carlo algorithm, whose convergence is rigorously studied in the paper as well.
Through extensive experiments on real-world classiﬁcation and regression problems, we show that the proposed method compares favorably with the existing state-of-the-art sufﬁcient dimension reduction methods and is computationally more efﬁcient for large-scale data.
Introduction
As a supervised method, sufﬁcient dimension reduction (SDR) aims to project the data onto a lower dimensional space so that the output is conditionally independent of the input features given the projected features.
Mathematically the problem of SDR can be described as follows.
Let Y ∈ R
d be the response variables and let X =
( X 1 , . . .
X p ) T ∈ R p be the explanatory variables of dimension p .
The goal of SDR is to ﬁnd a lower-dimensional representation Z ∈ R q , as a function of X for some q < p , such that
[Equation]
where | = denotes conditional independence.
Intuitively the deﬁnition (1) implies that Z has extracted all the information contained in X for predicting Y .
In the literature, SDR has been developed under both linear and nonlinear settings.
Under the linear setting, SDR is to ﬁnd a few linear combinations of X that are sufﬁcient to describe the conditional distribution of Y given X , i.e., ﬁnding a projection matrix B ∈ R p × q such that
[Equation]
A more general deﬁnition for linear SDR based on σ -ﬁeld can be found in [7].
Towards this goal, a variety of inverse regression methods have been proposed, see e.g., sliced inverse regression (SIR) Faming Liang [26], sliced average variance estimation (SAVE) [6, 8], parametric inverse regression [4], contour regression [25], and directional regression [24].
These methods require strict assumptions on the joint distribution of ( X , Y ) or the conditional distribution of X | Y , which limit their use in practice.
To address this issue, some forward regression methods have been developed in the literature, see e.g., principal Hessian directions [27], minimum average variance estimation [45], conditional variance estimation [12], among others.
These methods require minimal assumptions on the smoothness of the joint distribution ( X , Y ) , but they do not scale well for big data problems.
They can become infeasible quickly as both p and n increase, see [21] for more discussions on this issue.
Under the nonlinear setting, SDR is to ﬁnd a nonlinear function f ( · ) such that
[Equation]
A general theory for nonlinear SDR has been developed in [22].
A common strategy to achieve nonlinear SDR is to apply the kernel trick to the existing linear SDR methods, where the variable X is ﬁrst mapped to a high-dimensional feature space via kernels and then inverse or forward regression methods are performed.
This strategy has led to a variety of methods such as kernel sliced inverse regression (KSIR) [43], kernel dimension reduction (KDR) [13, 14], manifold kernel dimension reduction (MKDR) [35], generalized sliced inverse regression (GSIR) [22], generalized sliced average variance estimator (GSAVE) [22], and least square mutual information estimation (LSMIE) [42].
A drawback shared by these methods is that they require to compute the eigenvectors or inverse of an n × n matrix.
Therefore, these methods lack the scalability necessary for big data problems.
Another strategy to achieve nonlinear SDR is to consider the problem under the multi-index model setting.
Under this setting, the methods of forward regression such as those based on the outer product of the gradient [44, 20] have been developed, which often involve eigen-decomposition of a p × p matrix and are thus unscalable for high-dimensional problems.
Quite recently, some deep learning-based nonlinear SDR methods have been proposed in the literature, see e.g. [21, 2, 29], which are scalable for big data by training the deep neural network (DNN) with a mini-batch strategy.
In [21], the authors assume that the response variable Y on the predictors X is fully captured by a regression [Equation]
for an unknown function g ( · ) and a low rank parameter matrix B , and they propose a two-stage approach to estimate g ( · ) and B .
They ﬁrst estimate g ( · ) by ˜ g ( · ) by ﬁtting the regression Y = ˜ g
( X ) + (cid:15) with a DNN and initialize the estimator of B using the outer product gradient (OPG) approach [45], and then reﬁne the estimators of g ( · ) and B by optimizing them in a joint manner.
However, as pointed out by the authors, this method might not be valid unless the estimate of g ( · ) is consistent, but the consistency does not generally hold for the fully connected neural networks trained without constraints.
Speciﬁcally, the universal approximation ability of the DNN can make the latent variable Z :=
B T X unidentiﬁable from the DNN approximator of g ( · ) ; or, said differently, Z can be an arbitrary vector by tuning the size of the DNN to be sufﬁciently large.
A similar issue happened to [2], where the authors propose to learn the latent variable Z by optimizing three DNNs to approximate the distributions p ( Z | X ) , p ( X | Z ) and p ( Y | Z ) , respectively, under the framework of variational autoencoder.
Again, Z suffers from the identiﬁability issue due to the universal approximation ability of the DNN.
In [29], the authors employ a regular DNN for sufﬁcient dimension reduction, which works only for the case that the distribution of the response variable falls into the exponential family.
How to conduct SDR with DNNs for general large-scale data remains an unresolved issue.
We address the above issue by developing a new type of stochastic neural network.
The idea can be loosely described as follows.
Suppose that we are able to learn a stochastic neural network, which maps X to Y via some stochastic hidden layers and possesses a layer-wise Markovian structure.
Let h denote the number of hidden layers, and let Y 1 , Y 2 , . .
, Y h denote the outputs of the respective stochastic hidden layers.
By the layer-wise Markovian structure of the stochastic neural network, we can decompose the joint distribution of ( Y , Y h , Y h − 1 , . . .
, Y 1 ) conditioned on X as follows
[Equation]
where each conditional distribution is modeled by a linear or logistic regression (on transformed outputs of the previous layer), while the stochastic neural network still provides a good approximation to the underlying DNN under appropriate conditions on the random noise added to each stochastic layer.
The layer-wise Markovian structure implies Y | =
X | Y h , and the simple regression structure of π ( Y | Y h ) successfully gets around the identiﬁability issue of the latent variable Z :
=
Y h that has been suffered by some other deep learning-based methods [2, 21].
How to deﬁne and learn such a stochastic neural network will be detailed in the paper.
Our contribution in this paper is three-fold: (i)
We propose a new type of stochastic neural network (abbreviated as “StoNet” hereafter) for sufﬁcient dimension reduction, for which a layerwise Markovian structure (5) is imposed on the network in training and the size of the noise added to each hidden layer is calibrated for ensuring the StoNet to provide a good approximation to the underlying DNN.
(ii) We develop an adaptive stochastic gradient MCMC algorithm for training the StoNet and provides a rigorous study for its convergence under mild conditions.
The training algorithm is scalable with respect to big data and it is itself of interest to statistical computing for the problems with latent variables or missing data involved.
(iii)
We formulate the StoNet as a composition of many simple linear/logistic regressions, making its structure more designable and interpretable.
The backward imputation and forward parameter updating mechanism embedded in the proposed training algorithm enables the regression subtasks to communicate globally and update locally.
As discussed later, these two features enable the StoNet to solve many important scientiﬁc problems, rather than sufﬁcient dimension reduction, in a more convenient way than does the conventional DNN.
The StoNet bridges us from linear models to deep learning.
Other related works.
Stochastic neural networks have a long history in machine learning.
Famous examples include multilayer generative models [18], restricted Boltzmann machine [19] and deep
Boltzmann machine [38].
Recently, some researchers have proposed adding noise to the DNN to improve its ﬁtting and generalization.
For example, [39] proposed the dropout method to prevent the DNN from over-ﬁtting by randomly dropping some hidden and visible units during training; [32] proposed adding gradient noise to improve training; [16, 36, 47, 40] proposed to use stochastic activation functions through adding noise to improve generalization and adversarial robustness, and [48] proposed to learn the uncertainty parameters of the stochastic activation functions along with the training of the neural network.
However, none of the existing stochastic neural networks can be used for sufﬁcient dimension reduction.
It is known that the multilayer generative models [18], restricted Boltzmann machine [19] and deep Boltzmann machine [38] can be used for dimension reduction, but under the unsupervised mode.
As explained in [39], the dropout method is essentially a stochastic regularization method, where the likelihood function is penalized in network training and thus the hidden layer output of the resulting neural network does not satisfy (3).
In [16], the size of the noise added to the activity function is not well calibrated and it is unclear whether the true log-likelihood function is maximized or not.
The same issue happens to [32]; it is unclear whether the true log-likelihood function is maximized by the proposed training procedure.
In [36], the neural network was trained by maximizing a lower bound of the log-likelihood function instead of the true log-likelihood function; therefore, its hidden layer output does not satisfy (3).
In [47], the random noise added to the output of each hidden unit depends on its gradient; the mutual dependence between the gradients destroys the layer-wise Markovian structure of the neural network and thus the hidden layer output does not satisfy (3).
Similarly, in [48], independent noise was added to the output of each hidden unit and, therefore, the hidden layer output satisﬁes neither (5) nor (3).
In [40], inclusion of the support vector regression (SVR) layer to the stochastic neural network makes the hidden layer outputs mutually dependent, although the observations are mutually independent.
StoNet for Sufﬁcient Dimension Reduction
In this section, we ﬁrst deﬁne the StoNet, then justify its validity as a universal learner for the map from X to Y by showing that the StoNet has asymptotically the same loss function as a DNN under appropriate conditions, and further justify its use for sufﬁcient dimension reduction.
2.1 The StoNet
Consider a DNN model with h hidden layers.
For the sake of simplicity, we assume that the same activation function ψ is used for each hidden unit.
By separating the feeding and activation operators of each hidden unit, we can rewrite the DNN in the following form
[Equation]
where e h +1 ∼ N (0 , σ 2 h
+1 I
d h +1 ) is Gaussian random error;
˜ Y i , b i ∈ R
d i for i = 1 , 2 , . . .
, h ;
Y , b h +1 ∈ R d
[Equation]
1 ; Ψ( ˜ Y i − [Equation] ) = ( ψ ( ˜ Y
i [Equation] ) , ψ ( ˜ Y i − 1 , 2 ) , . . , ψ ( ˜ Y i −
[Equation] d i − 1 )) T for i = 2 , 3 , . .
, h + 1 , ψ ( · ) is the activation function, and
˜ Y
i − 1 ,j is the j th element of ˜ Y i − 1 ;
w i ∈ R d i × d i − 1 for i = 1 , 2 , . . .
, h + 1 , and d 0 = p denotes the dimension of X .
For simplicity, we consider only the regression problems in (6).
By replacing the third equation in (6) with a logit model, the DNN can be trivially extended to the classiﬁcation problems.
The StoNet, as a probabilistic deep learning model, can be constructed by adding auxiliary noise to ˜ Y
i ’s, i = 1 , 2 , . . .
, h in (6).
Mathematically, the StoNet is given by
[Equation]
where Y 1 , Y 2 , . .
, Y h can be viewed as latent variables.
Further, we assume that e i ∼ N (0 , σ {+
+} i I d i ) for i = 1 , 2 , . . .
, h, h + 1 .
For classiﬁcation networks, the parameter σ 2 h +1 plays the role of temperature for the binomial or multinomial distribution formed at the output layer, which works with { σ 21 , . . .
, σ 2 h } together to control the variation of the latent variables { Y 1 , . .
, Y h } .
Figure 1 depicts the architecture of the StoNet.
In words, the StoNet has been formulated as a composition of many simple linear/logistic regressions, which makes its structure more designable and interpretable.
Refer to Section 5 for more discussions on this issue.
2.2 The StoNet as an Approximator to a DNN
To show that the StoNet is a valid approximator to a DNN, i.e., asymptotically they have the same loss function, the following conditions are imposed on the model.
To indicate their dependence on the training sample size n , we rewrite σ i as σ n,i for i = 1 , 2 , . .
, h
+ 1 .
Let θ i = ( w i , b i ) , let θ =
( θ 1 , θ 2 · · · , θ h +1 ) denote the parameter vector of StoNet, let d θ denote the dimension of θ , and let Θ denote the space of θ .
Assumption A1 (i) Θ is compact, i.e., Θ is contained in a d θ -ball centered at 0 with radius r ;
(ii) E (log π ( Y | X , θ )) 2 < ∞ for any θ ∈ Θ ; (iii) the activation function ψ ( · ) is c (cid:48)
-Lipschitz continuous for some constant c (cid:48) ; (iv)
the network’s depth h and widths
d i ’s are both allowed to increase with n ; (v) σ n, 1 ≤ σ n, 2 ≤ · · · ≤ σ n,h +1 , σ n,h[-+
-]
+1 = O (1) , and d h +1 ( (cid:81) hi =
k +1 d 2 i )
d k σ 2 n,k ≺ 1 h for any k ∈ { 1 , 2 , . . .
, h } .
Condition (i) is more or less a technical condition.
As shown in Lemma S1 (in supplementary material), the proposed training algorithm for the StoNet ensures the estimates of θ to be L 2 -upper bounded.
Condition (ii) is the regularity condition for the distribution of Y .
Condition (iii) can be satisﬁed by many activation functions such as tanh , sigmoid and ReLU .
Condition (v) constrains the size of the noise added to each hidden layer such that the StoNet has asymptotically the same loss function as the DNN when the training sample size becomes large, where the factor d h +1 ( (cid:81)
hi
=
k +1 d 2 i )
d k is derived in the proof of Theorem 2.1 and it can be understood as the ampliﬁcation factor of the noise e k at the output layer.
Let L : Θ → R denote the loss function of the DNN as deﬁned in (6), which is given by
[Equation]
where n denotes the training sample size, and i indexes the training samples.
Theorem 2.1 shows that the StoNet and the DNN have asymptotically the same training loss function.
Theorem 2.1 Suppose Assumption A1 holds.
Then the StoNet (7) and the neural network (6) have asymptotically the same loss function, i.e.,
[Equation]
where Y mis =
( Y 1 , Y 2 , . . .
, Y h ) denotes the collection of all latent variables in the StoNet (7).
Let Q ∗ ( θ ) =
E (log π ( Y | X , θ )) , where the expectation is taken with respect to the joint distribution π ( X , Y ) .
By Assumption A1- ( i )&( ii ) and the law of large numbers,
[Equation]
holds uniformly over Θ .
Further, we assume the following condition hold for Q ∗ ( θ ) :
Assumption A2 (i) Q ∗ ( θ ) is continuous in θ and uniquely maximized at θ ∗ ; (ii) for any (cid:15) > 0 , sup θ ∈ Θ \ B ( (cid:15) )
Q ∗ ( θ ) exists, where B ( (cid:15) )
= { θ : (cid:107) θ − θ ∗ (cid:107) <
(cid:15) } , and δ =
Q ∗ ( θ ∗ ) − sup θ ∈ Θ \ B ( (cid:15) )
Q ∗ ( θ ) > 0 .
Assumption A2 is more or less a technical assumption.
As shown in [34] (see also [15]), for a fully connected DNN, almost all local energy minima are globally optimal if the width of one hidden layer of the DNN is no smaller than the training sample size and the network structure from this layer on is pyramidal.
Similarly, [1], [11], [50], and [49] proved that the gradient-based algorithms with random initialization can converge to the global optimum provided that the width of the DNN is polynomial in training sample size.
All the existing theory implies that this assumption should not be a practical concern for StoNet as long as its structure is large enough, possibly over-parameterized, such that the data can be well ﬁtted.
Further, we assume that each θ for the DNN is unique up to loss-invariant transformations, such as reordering some hidden units and simultaneously changing the signs of some weights and biases.
Such an implicit assumption has often been used in theoretical studies for neural networks, see e.g. [28] and [41] for the detail.
Theorem 2.2 Suppose Assumptions A1 and A2 hold, and π ( Y , Y mis | X , θ ) is continuous in θ .
Let ˆ θ n = arg max [Equation] [Equation] n (cid:80)
[Equation] =
[Equation] Y [Equation] i ) , Y
[Equation]
i ) mis | X ( i ) , θ ) } .
Then (cid:107) ˆ θ n [Equation] θ ∗ (cid:107) p → 0 as n → ∞ .
This theorem implies that the DNN (6) can be trained by training the StoNet (7), which are asymptotically equivalent as the sample size n becomes large.
Refer to the supplement for its proof.
2.3 Nonlinear Sufﬁcient Dimension Reduction via StoNet
The joint distribution π ( Y , Y mis | X , θ ) for the StoNet can be factored as [Equation]
[Equation]
based on the Markovian structure between layers of the StoNet.
Therefore
[Equation]
By Proposition 2.1 of [23], Equation (12) is equivalent to Y | =
X | Y h , which coincides with the deﬁnition of nonlinear sufﬁcient dimension reduction in (3).
In summary, we have the proposition:
Proposition 2.1 For a well trained StoNet for the mapping X
→ Y , the output of the last hidden layer Y h satisﬁes SDR condition in (3).
The proof simply follows the above arguments and the properties of the StoNet.
Proposition 2. implies that the StoNet can be a useful and ﬂexible tool for nonlinear SDR.
However, the conventional optimization algorithm such as stochastic gradient descent (SGD) is no long applicable for training the StoNet.
In the next section, we propose to train the StoNet using an adaptive stochastic gradient
MCMC algorithm.
At the end of the paper, we discuss how to determine the dimension of Y h via regularization at the output layer of the StoNet.
An Adaptive Stochastic Gradient MCMC algorithm 3.1 Algorithm Establishment
Adaptive stochastic gradient MCMC algorithms have been developed in [10] and [9], which work under the framework of stochastic approximation MCMC [3].
Suppose that we are interested in solving the mean ﬁeld equation
[Equation]
where π ( Z | θ ) denotes a probability density function parameterized by θ .
The adaptive stochastic gradient MCMC algorithm works by iterating between the steps: (i) sampling , which is to generate a Monte Carlo sample Z ( k ) from a transition kernel that leaves π ( Z | θ ( k ) ) as the equilibrium distribution; and (ii) parameter updating , which is to update θ ( k ) based on the current sample Z ( k ) in a stochastic approximation scheme.
These algorithms are said “adaptive” as the transition kernel used in step (i) changes with iterations through the working estimate θ ( k ) .
By Theorem 2.2, the StoNet can be trained by solving the equation
[Equation]
[Equation]
where H ( Y mis , θ )
=
∇ θ log π ( Y , Y mis | X , θ ) .
Applying the adaptive stochastic gradient MCMC algorithm to (14) leads to Algorithm 1, where stochastic gradient Hamilton Monte Carlo (SGHMC) [5] is used for simulating the latent variables Y mis .
Algorithm 1 is expected to outperform the basic algorithm by [10], where SGLD is used in the sampling step, due to the accelerated convergence of
SGHMC over SGLD [33].
In Algorithm 1, we let ( Y ( s.k ), Y ( s.k ) h +1 ) =
( X ( s ) , Y ( s ) ) denote a training sample s , and let Y ( s.k ) mis
=
( Y ( s.k ), . . .
, Y ( s.k ) h ) denote the latent variables imputed for the training sample s at iteration k .
To make the computation for the StoNet scalable with respect to the training sample size, we train the parameter θ with mini-batch data and then extract the SDR predictor Y h with the full dataset; that is, we can run Algorithm 1 in two stages, namely, θ -training and SDR.
In the θ -training stage, the algorithm is run with mini-batch data until convergence of θ has been achieved; and in the SDR stage, the algorithm is run with full data for a small number of iterations.
In this paper, we typically set the number of iterations/epochs of the SDR stage to 30.
The proposed algorithm has the same order of computational complexity as the standard SGD algorithm, although it can be a little slower than SGD due to multiple iterations being performed at each backward sampling step.
3.2 Convergence Analysis of Algorithm
Notations: We let D =
( D 1 , D 2 , . . .
, D n ) denote a dataset of n observations.
For StoNet, D i has included both the input and output variables of the observation.
We let Y mis =
Z = ( Z 1 , Z 2 , . .
, Z n ) , where Z i is the latent variable corresponding to D i , and let f D i
( z i , θ )
= − log π ( z i | D i , θ ) .
Let z =
( z 1 , z 2 , . . .
, z n ) be a realization of ( Z 1 , Z 2 , . .
, Z n ) , and let F D ( Z , θ )
= (cid:80) ni
=
f D i
( z i , θ ) .
For simplicity, we assume (cid:15) k,i = (cid:15) k for i
= 1 , 2 , . .
, h , and γ k,i = γ k for i
= 1 , 2 , . .
, h + 1 .
To facilitate theoretical study, one iteration of Algorithm 1 is rewritten in the new notations as follows. i) ( Sampling ) Simulate the latent variable Z by setting
[Equation]
where η is the friction coefﬁcient, β is the inverse temperature, k indexes the iteration, (cid:15) k
+ is the learning rate, and ∇ Z ˆ F D ( Z ( k ) , θ ( k ) ) is an estimate of ∇ Z F D ( Z ( k ) , θ ( k ) ) .
(ii) ( Parameter updating ) Update the parameters θ by setting
[Equation]
where γ k +1 is the step size, H ( Z ( k +1) , θ ( k ) )
[Equation]
and S
[Equation]
denotes a minibatch of the full dataset.
For each observation s ∈ S k , sample Y i ’s in the order from layer h to layer 1 .
More explicitly, we sample Y ( s,k ) i from the distribution
[Equation]
by running SGHMC in t HMC steps:
Initialize v ( s, 0) i = 0 , and initialize Y ( s,k, 0) i by the corresponding ˜ Y i calculated in (6).
for l
, t HMC do for i h, h − 1 ,
, 1 do
[Equation]
| ˆ θ ( k − 1) i , Y ( s,k,l − 1) i −(cid:17)
+ (cid:15) k,i ∇ Y ( s,k,l − 1)
i log π (cid:16) Y ( s,k,l − 1) i
+ | ˆ θ ( k − 1)
i +1 , Y ( s,k,l − 1)
i (cid:17) + (cid:112) 2 (cid:15) k,i η e ( s,k,l ) ,
Y ( s,k,l ) i =
Y ( s,k,l − 1) i + (cid:15) k,i v ( s,k,l − 1) i , (15) where e s,k,l ∼ N (0 , I d i ) , (
cid:15) k,i is the learning rate, and η is the friction coefﬁcient.
end end
Set Y ( s,k ) i =
Y ( s,k,t HMC ) i for i
= 1 , 2 , . .
, h .
STEP 2: Parameter Update
Update the estimates of ˆ θ ( k − 1) =
(ˆ θ ( k − 1), ˆ θ ( k − 1), . . .
, ˆ θ ( k − 1) h +1 ) by ˆ θ ( k ) i
= ˆ θ ( k − 1)
i + γ k,i n | S k | (cid:88) s ∈ S k ∇
θ
i log π ( Y ( s,k ) i | ˆ θ ( k − 1)
i , Y ( s,k ) i − 1 ) , i = 1 , 2 , . . .
, h + 1 , where γ k,i is the step size used for updating θ i .
end
Theorem 3.1 Suppose Assumptions B1-B6 (in the supplementary material) hold.
If we set (cid:15) k = C (cid:15) / ( c e + k α ) and γ k = C γ / ( c g + k α ) for some constants α ∈ (0 , 1) , C (cid:15) > 0 , C γ > 0 , c e ≥ 0 and c g ≥ 0 , then there exists an iteration k 0 and a constant λ 0 > 0 such that for any k > k 0 ,
[Equation]
where θ ∗ denotes a solution to the equation (14), and the explicit form of λ 0 is given in Theorem S1.
Let µ D ,k denote the probability law of ( Z ( k ) , v ( k ) ) given the dataset D , let π D denote the target distribution
π ( z | θ ∗ , D ) , let T k
= (cid:80) k − 1 i =
(cid:15) i +1 , and let H ρ ( · , · ) denote a semi-metric for probability distributions.
Theorem 3.2 establishes convergence of µ D ,k .
Theorem 3.2 Suppose Assumptions B1-B7 (in the supplementary material) hold.
Then for any k ∈ N ,
[Equation]
which can be made arbitrarily small by choosing a large enough value of T k and small enough values of (cid:15) 1 and γ 1 , provided that { (cid:15) k } and { γ k } are set as in Theorem S1.
Here C 5 and C 6 denote some constants and an explicit form of ˜ C ( k ) is given in Theorem S2.
As implied by Theorem 3.2, Z ( k ) converges weakly to the distribution π
( Z | θ ∗ , D ) as k → ∞ , which ensure validity of the decomposition (11) and thus the followed SDR.
We note that our theory is very different from [10].
First, Θ is essentially assumed to be bounded in [10], while our study is under the assumption Θ = R d θ .
Second, for weak convergence of latent variables, only the convergence of the ergodicity average is studied in [10], while we study their convergence in 2-Wasserstein distance such that (11) holds and SDR can be further applied.
Numerical Studies
In this section, we empirically evaluate the performance of the StoNet on SDR tasks.
We ﬁrst compare the StoNet with an existing deep SDR method, which validates the StoNet as a SDR method. Then we compare the StoNet with some existing linear and nonlinear SDR methods on classiﬁcation and regression problems. For each problem, we ﬁrst apply the StoNet and the linear and nonlinear SDR methods to project the training samples onto a lower dimensional subspace, and then we train a separate classiﬁcation/regression model with the projected training samples. A good SDR method is expected to extract the response information contained in the input data as much as possible.
The example for multi-label classiﬁcation is presented in the supplement.
We use the M1 example in [21] to illustrate the non-identiﬁability issue suffered by the deep SDR methods developed in [21] and [2].
The dataset consists of 100 independent observations. Each observation is generated from the model Y = cos( X T b ) + (cid:15) , where X ∈ R 20 follows a multivariate
Gaussian distribution, b ∈ R 20 is a vector with the ﬁrst 6 dimensions equal to 1 √ 6 and the other dimensions equal to 0, and (cid:15) follows a generalized Gaussian distribution GN (0 , (cid:112) 1 / 2 , 0 . 5) . We use the code 2 provided by [21] to conduct the experiment, which projects the data to one-dimensional space by working with a reﬁnement network of structure 20-1-512-1. Let Z 1 and Z 2 denote two SDR vectors produced by the method in two independent runs with different initializations of network weights.
We then test the independence of Z 1 and Z 2 using the R package RCIT 3 .
The test returns a p -value of 0.4068, which does not reject the null hypothesis that Z 1 and Z 2 are independent.
We have also applied the proposed method to the same dataset, where the StoNet has a structure of 20-10-1-1 and tanh is used as the activation function.
In this way, the StoNet projects the data to onedimensional space.
The SDR vectors produced in two independent runs (with different initializations of network weights) of the method are collected and tested for their dependence.
The test returns a p -value of 0.012, which suggests that the two SDR vectors are not independent.
This example suggests that if a complicated neural network model is used to ﬁt the function g ( · ) in (4), then the dimension reduced data do not necessarily capture the information of the original data.
We ﬁrst test the StoNet on some binary classiﬁcation examples taken from [37] and [46].
The dimensions of these examples are generally low, ranging from 6 to 21.
There are two steps for each example: ﬁrst, we project the training samples onto a low-dimensional subspace with dimension q = (cid:98) p/ 2 (cid:99) or q = (cid:98) p/ 4 (cid:99) and then train a logistic regression model on the projected predictors for the binary classiﬁcation task.
We trained two one-hidden-layer StoNets with (cid:98) p/ 2 (cid:99) and (cid:98) p/ 4 (cid:99) hidden units, respectively.
For comparison, four state-of-the-art non-linear SDR methods, including LSMIE, GSIR, GSAVE and KDR 4 , were trained to extract nonlinear sufﬁcient predictors.
In addition, three popular linear dimension reduction methods were taken as baselines for comparison, which include SIR, SAVE and PCA 5 .
The hyperparameters of these methods were determined with 5- cross-validation in terms of misclassiﬁcation rates.
Random partitioning of the dataset in crossvalidation makes their results slightly different in different runs, even when the methods themselves are deterministic.
Refer to the supplement for the parameter settings used in the experiments.
The results are summarized in Table 1, which reports the mean and standard deviation of the misclassiﬁcation rates averaged over 20 independent trials.
Table 1 shows that StoNet compares favorably to the existing linear and nonlinear dimension reduction methods.
[Table]
We have also tested the methods on a multi-label classiﬁcation problem with a sub-MNIST dataset.
Refer to Section 1 (of the supplement) for the details of the experiments.
The numerical results are summarized in Table
[Table]
which indicates the superiority of SToNet over the existing nonlinear SDR methods in both computational efﬁciency and prediction accuracy. [Table]
4.3 A Regression Example
The dataset, relative location of CT slices on axial axis 6 , contains 53,500 CT images collected from 74 patients.
There are 384 features extracted from the CT images [Table] and the response is the relative location of the CT slice on the axial axis of the human body which is between 0 and 180 (wheredenotes the top of the head and 180 the soles of the feet).
Our goal is to predict the relative position of CT slices with the high-dimensional features.
Due to the large scale of the dataset and the high computation cost of the nonlinear SDR methods
LSMIE, GSIR and GSAVE, we don’t include them as baselines here.
Similar to the previous examples, the experiment was conducted in two stages.
First, we applied the dimension reduction methods to project the data onto a lower-dimensional subspace, and then trained a DNN on the projected data for making predictions.
Note that for the StoNet, the dimension reduced data can be modeled by a linear regression in principle and the DNN is used here merely for fairness of comparison; while for autoencoder and PCA, the use of the DNN for modeling the dimension reduced data seems necessary for such a nonlinear regression problem.
The mean squared error (MSE) and Pearson correlation were used as the evaluation metrics to assess the performance of prediction models.
For this example, we have also trained a DNN with one hidden layer and 100 hidden units as the comparison baseline.
Refer to the supplementary material for the hyperparameter settings used in the experiments.
[Table]
The results, which are summarized in Figure S2 (in the supplement) and Table 3, show that as the dimension q decreases, the performance of Autoencoder degrades signiﬁcantly.
In contrast, StoNet can achieve stable and robust performance even when q is reduced to 3.
Moreover, for each value of q , StoNet outperforms Autoencoder and PCA signiﬁcantly in both MSE and Pearson correlation.
Conclusion
In this paper, we have proposed the StoNet as a new type of stochastic neural network under the rigorous probabilistic framework and used it as a method for nonlinear SDR.
The StoNet, as an approximator to neural networks, possesses a layer-wise Markovian structure and SDR can be obtained by extracting the output of its last hidden layer.
The StoNet overcomes the limitations of the existing nonlinear SDR methods, such as inability in dealing with high-dimensional data and computationally inefﬁciency for large-scale data.
We have also proposed an adaptive stochastic gradient MCMC algorithm for training the StoNet and studied its convergence theory.
Extensive experimental results show that the StoNet method compares favorably with the existing state-of-the-art nonlinear SDR methods and is computationally more efﬁcient for large-scale data.
In this paper, we study SDR with a given network structure under the assumption that the network structure has been large enough for approximating the underlying true nonlinear function.
To determine the optimal depth, layer width, etc., we can combine the proposed method with a sparse deep learning method, e.g. [17] and [41].
That is, we can start with an over-parameterized neural network, employ a sparse deep learning technique to learn the network structure, and then employ the proposed method to the learned sparse DNN for the SDR task.
We particularly note that the work [41] ensures consistency of the learned sparse DNN structure, which can effectively avoid the over sufﬁciency issue, e.g., learning a trivial relation such as identity in the components.
As an alternative way to avoid the over sufﬁciency issue, we can add a post sparsiﬁcation step to the StoNet, i.e., applying a sparse SDR procedure (e.g., [30] and [31] with a Lasso penalty) to the output layer regression of a learnt StoNet by noting that the regression formed at each node of the StoNet is a multiple index model. In this way, the StoNet, given its universal approximation ability, provides a simple method for determining the central space of SDR for general nonlinear regression models. The StoNet has great potentials in machine learning applications.
Like other stochastic neural networks [16, 36, 47, 40], it can be used to improve generalization and adversarial robustness of the
DNN.
The StoNet can be easily extended to other neural network architectures such as convolutional neural network (CNN), recurrent neural network (RNN) and Long Short-Term Memory (LSTM) networks.
For CNN, the randomization technique (7) can be directly applied to the fully connected layers.
The same technique can be applied to appropriate layers of the RNN and LSTM as well. Acknowledgments
Liang’s research is support in part by the NSF grants DMS-2015498 and DMS-2210819, and the
NIH grant R01-GM126089.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] (c) Did you discuss any potential negative societal impacts of your work?
[N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[N/A] (b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]