P ROBABILISTIC MODELING THE HIDDEN LAYERS OF D EEP N EURAL N ETWORKS
Anonymous authors
Paper under double-blind review
A BSTRACT
In this paper, we demonstrate that the parameters of Deep Neural Networks (DNNs) do not satisfy the i.i.d.
prior assumption and activations being i.i.d. is not valid for all the hidden layers of DNNs.
Hence, the Gaussian Process cannot correctly explain all the hidden layers of DNNs.
Alternatively, we introduce a novel probabilistic representation for the hidden layers of DNNs in two aspects: (i) a hidden layer formulates a Gibbs distribution, in which neurons deﬁne the energy function, and (ii) the connection between two adjacent layers can be modeled by a product of experts model.
Based on the probabilistic representation, we demonstrate that the entire architecture of DNNs can be explained as a Bayesian hierarchical model.
Moreover, the proposed probabilistic representation indicates that DNNs have explicit regularizations deﬁned by the hidden layers serving as prior distributions.
Based on the Bayesian explanation for the regularization of
DNNs, we propose a novel regularization approach to improve the generalization performance of DNNs.
Simulation results validate the proposed theories.
I NTRODUCTION
Recently, interpreting the hidden layers of Deep Neural Networks (DNNs) as the Gaussian Process (GP) has attracted a great deal of attention because it provides a novel probabilistic perspective to clarify the working mechanism of deep learning.
Neal (1994) initially demonstrates the equivalence between GP and neural networks with a single fully connected layer in the limit of inﬁnite neurons.
Following this seminal work, Lee et al.
; Matthews et al.
(2018) further extend the equivalence to neural networks with multiple hidden layers and Garriga-Alonso et al.
; Novak et al.
(2018) establish the correspondence between the convolutional layers and GP.
It is important to note that the GP explanations rely on a fundamental probabilistic premise that the activations of a hidden layer are independent and identically distributed (i.i.d.).
More speciﬁcally, based on the classical Central Limit Theorem (CLT), the prerequisite of an activation of a hidden layer approaching a Gaussian distribution is that all the activations of the previous layer are i.i.d..
Though the Lyapunov CLT could relax the probabilistic premise to be independence only, all the previous works do not thoroughly discuss the probabilistic premise except assuming an i.i.d.
prior for all the parameters of DNNs.
Due to its extreme importance but unclear status, it is necessary to thoroughly investigate the probabilistic premise for improving the interpretability of DNNs.
In this work, we demonstrate that the parameters of DNNs are correlated and depend on the training dataset, i.e., they cannot satisfy the i.i.d.
prior.
Moreover, we demonstrate that activations being i.i.d.
cannot hold for all the hidden layers of DNNs.
In the context of Bayesian probability, we theoretically derive the necessary conditions for the activations of a hidden layer being i.i.d.
given the assumption that the activations of the previous layer are i.i.d..
Subsequently, we experimentally show that typical DNNs, such as the Multilayer Perceptron (MLP) and the Convolutional Neural
Networks (CNNs), cannot satisfy the necessary conditions.
As a result, activations being i.i.d. is not valid for all hidden layers.
In other words, GP with the i.i.d.
prior cannot correctly explain all the hidden layers of DNNs.
In addition, some previous works show that GP is sensitive to the curse of dimensionality (Bengio et al., 2005; Hinton et al., 2012) and cannot clarify the hierarchical representation, an essential of deep learning (Matthews et al., 2018).
Alternatively, we propose a novel probabilistic representation for the hidden layers based on the
Gibbs distribution (LeCun et al., 2006) and the Product of Experts (PoE) model (Hinton, 2002).
The proposed probabilistic representation explains a hidden layer in two aspects: (i) a hidden layer can be formulated as a Gibbs distribution, in which neurons deﬁne the energy function, and (ii) the connection between two adjacent layers can be modeled by a PoE model.
Speciﬁcally, the distribution of a single neuron or hidden unit (e.g., convolutional channel) can be expressed as a
PoE, in which all the experts are deﬁned by the neurons or hidden units in the previous hidden layer.
Compared to the GP explanations, the proposed probabilistic representation provides a more speciﬁc explanation for the hidden layers of DNNs without any assumption.
Based on the probabilistic representation, the entire architecture of DNNs can be explained as a
Bayesian Hierarchical Model (BHM) (Li & Perona, 2005).
Precisely, the output layer formulates a likelihood distribution, the hidden layers recursively serve as prior distributions for their next layers, and the ﬁrst hidden layer formulates a prior distribution of the training dataset.
In contrast to previous works claiming that the backpropagation training performs implicit regularization for
DNNs (Zhang et al., 2016; Neyshabur et al., 2017), the BHM explanation indicates that DNNs have explicit regularizations because prior distributions correspond to regularizations in the context of
Bayesian regularization theory (Steck & Jaakkola, 2003).
Based on the BHM explanations for the architecture of DNNs, we propose a novel regularization approach to improve the generalization performance of DNNs through pre-training the hidden layers corresponding to prior distributions.
B ACKGROUND
Probabilistic modeling plays an extremely important role in machine learning.
Most structured probabilistic models can be formulated as a Gibbs distribution, which describes the dependence of random variables x by associating an energy to each dependence structure.
[Equation]
, (1) where E ( x ; θ ) is the energy function, Z ( θ )
= (cid:80) x exp [ − E ( x ; θ )] is the partition function, and θ denote all the parameters.
It is important to note that E ( x ; θ ) is the sufﬁcient statistics for p ( x ; θ ) , i.e., p ( x ; θ ) entirely depends on E ( x ; θ ) because Z ( θ ) only relies on θ (Cover &
Thomas, 2006).
An important Gibbs distribution is the Markov Random Fields (MRFs) and the corresponding energy function is equivalent to the summation of multiple potential functions, i.e.,
E ( x ; θ ) = − (cid:80) k f k ( x ; θ k ) (Geman & Geman, 1984; Wang et al., 2013; Goodfellow et al., 2016).
Assuming a single potential function deﬁnes an expert, i.e., F k = 1 Z ( θ k ) exp [ − f k ( x ; θ k )] , we can view MRFs as a PoE model, i.e., p ( x ; θ ) =
1 Z ( θ ) (cid:81)
k F k , where Z ( θ ) = (cid:81)
k Z ( θ k ) (Hinton, 2002).
Recently, numerous efforts have been devoted to explaining DNNs as a structured probabilistic model.
These explanations include Boltzmann distribution (Salakhutdinov & Hinton, 2009; Mehta & Schwab, 2014), random ﬁled theory (Zheng et al., 2015), mixture model (Oord & Schrauwen, 2014; Tang et al., 2015; Patel et al., 2016) and Gaussian process (Williams, 1997; Borovykh, 2018).
M AIN RESULTS 3. P RELIMINARIES
We assume that X and Y are two random variables and P θ ( X , Y )
=
P ( Y | X ) p ( X ) is an unknown joint distribution between X and Y , where P ( X ) describes the prior knowledge of X , P ( Y | X ) describes the statistical connection between X and Y , and θ indicate the parameters of P θ ( X , Y ) .
A dataset D = { ( x j , y j ) | x j ∈ R M , y j ∈ R L } J j =1 is composed of i.i.d.
samples generated from
P θ ( X , Y ) .
A neural network with I hidden layers is denoted as DNN = { x ; f 1 ; ... ; f I ; f Y } and trained by D , where ( x , y ) ∈
D are the input of the DNN and the corresponding training label, and f Y is an estimation of the true distribution P ( Y | X ) .
As a result, x ∼ P ( X ) and f Y ≈ P
( Y | X ) .
The meaning of f i is two-fold: (i) denoting all the neurons in the i th hidden layer, and (ii) representing the activations of the i th hidden layer.
In addition, F i and F Y are the random variables corresponding to f i and f Y , respectively.
𝑓 "" 𝑥 " 𝒙 𝑥 % 𝑥 & 𝑓 "’ 𝑓 "( 𝑓 ) (1) 𝛽 ’" 𝛽 "" 𝛽 (" 𝛼 "’ 𝛼 %’ 𝛼 &’ 𝑓 "/ 𝛽 /" ⋮ ⋮ ⋮ ⋮ 𝒇𝒇 𝒀 𝑓 ) (𝐿) 𝑓 ’" 𝑓 ’’ 𝑓 ’ ⋮ 𝒇𝛾 "8 𝛾 ’ 𝛾
3. T HE PARAMETERS OF DNN S CANNOT SATISFY THE I . I . D . ASSUMPTION
Above all, we examine the statistical properties of the parameters of DNNs on the benchmark
MNIST and CIFAR-10 datasets and show the parameters of DNNs being correlated to each other.
Since DNNs have so many parameters, it is very difﬁcult to directly show the correlation of different parameters.
Alternatively, we derive a necessary condition for the parameters of DNNs satisfying the i.i d. prior and experimentally show that typical DNNs cannot satisfy the necessary condition.
Therefore, the parameters of DNNs cannot satisfy the i.i.d. assumption.
If all the parameters of DNNs are i.i.d., the parameters of each neuron are identically distributed, e.g., { α mn } Mm =1 ∼ P ( α n ) in the MLP (Figure 1).
Since all the parameters are i.i.d., different neurons should be uncorrelated, i.e., ∀ n (cid:54) = n (cid:48) , Corr ( α n , α n (cid:48) )
= 0 .
In particular, the i.i.d.
assumption enables us to use the sample correlation r ( α n , α n (cid:48) ) to estimate Corr ( α n , α n (cid:48) ) .
[Equation]
where α n = 1 M (cid:80) Mm =1 α mn is the sample mean of { α mn } Mm =1 .
Therefore, taking into account the estimation error, | r ( α n , α n (cid:48) ) | close to zero is the necessary condition for all parameters being i.i.d..
First, we design a neural network with a single fully connected hidden layer, i.e., NN = { x ; f ; f Y } , to classify the MNIST dataset based on the seminal work (Neal, 1994).
After training the NN well, we calculate the absolute sample correlations between the weights of different neurons in f .
The result shows that the weights do not satisfy the necessary condition in Appendix A.1.
As a result, the parameters of the NN = { x ; f ; f Y } cannot satisfy the i.i.d assumption.
Second, we use a DNN (i.e., the above MLP) to classify the same dataset.
The information about the network architecture and training methods is included in Appendix A.2.
After training the MLP well, we calculate the absolute sample correlations, i.e., | r ( α n , α n (cid:48) ) | , | r ( β k , β k (cid:48) ) | , and | r ( γ l , γ l (cid:48) )
| , and derive three absolute correlation matrices, i.e., A (cid:48)
N × N , B (cid:48)
K × K , and C (cid:48)
L × L , shown in Figure 2.
We ﬁnd that most | r ( α n , α n (cid:48) ) | are close to zero, but most | r ( β k , β k (cid:48) )
| and | r ( γ l , γ l (cid:48) )
| are non-zero.
That indicates the necessary condition cannot hold for all the parameters of the MLP.
In addition, we construct a CNN for classifying the CIFAR-10 dataset and examine the necessary condition for the CNN in Appendix A.3, which shows similar results as the MLP.
Third, we visualize all the parameters of the NN = { x ; f ; f Y } for classifying a synthetic dataset in Appendix A.4.
Since the synthetic dataset is much simpler than MNIST, we can directly show the correlation of different parameters and their dependences on the dataset by visualizing them.
Overall, these simulations demonstrate that the parameters of DNNs cannot be i.i.d..
[Figure]
3. A CTIVATIONS BEING I . I . D . IS NOT VALID FOR ALL THE HIDDEN LAYERS OF DNN S
Although we demonstrate that the parameters of DNNs cannot satisfy the i.i.d.
assumption, we are still unclear whether the activations of a hidden layer are i.i.d..
In this section, we demonstrate that activations being i.i.d.
is not valid for all the hidden layers of DNNs.
In the context of Bayesian probability, all the weights and biases of the MLP (Figure 1) have prior distributions, i.e., α mn ∼ P ( A mn ) , β nk ∼ P
( W nk ) , b 1 n ∼ P
( B 1 n ) , and b 2 k ∼ P ( B 2 k ) , thus we regard G 1 n = (cid:80)
Mm =
A mn X m + B 1 n as the random variable of g 1 n ( x ) = (cid:80)
Mm =1 α mn · x m + b 1 n , and G 2 k
= (cid:80) Nn
=1 W nk F 1 n + B 2 k as the random variable of g 2 k ( f 1 ) =
(cid:80) Nn =1 β nk · f 1 n + b 2 k .
Hence, the random variables of two arbitrary activations in f 1 and f 2 , e.g., f 1 n and f 2 k , can be expressed as F 1 n = σ 1 ( G 1 n ) and F 2 k = σ 2 ( G 2 k ) , respectively.
Given two random variables A and B , the necessary conditions for them being i.i.d.
are that they are uncorrelated (i.e., Cov ( A, B ) = 0 ) and have the same expectation (i.e., E ( A )
=
E ( B ) ).
Similarly, assuming { F 1 n } Nn =1 are i.i.d.
and { f 1 n } Nn =1 ∼ P ( F 1 ) , we derive the following necessary conditions for the activations of the second hidden layer, i.e. , { F 2 k } Kk =1 , being i.i.d..
[Equation]
≤ K } , we must have
[Equation]
σ 2 ( · ) is strictly increasing, differentiable, and invertible.
[Equation]
where W k = (cid:80) Nn =1 W nk .
Equation 3 and 4 are the necessary conditions for { G 2 k } Kk =
being i.i.d..
Equation 5 constrains the activation function σ 2 ( · ) to guarantee that if { G 2 k } Kk =1 are i.i.d.
then { F 2 k } Kk =1 are i.i.d.
as well.
The necessary conditions are valid for arbitrary fully connected layers as long as properly changing the subscripts.
The detailed derivations are included in Appendix B and
Appendix C. Moreover, we prove that the necessary conditions also hold for convolutional layers based on the connection between fully connected layers and convolutional layers in Appendix E.
It is noteworthy that the i.i.d.
assumption for the parameters of DNNs indeed satisfy the necessary conditions for activations being i.i.d..
Nevertheless, since the i.i.d.
prior is not an appropriate prior for the parameters of DNNs, it cannot guarantee activations of a hidden layer being i.i.d..
Alternatively, the theoretical necessary conditions indicate an applicable approach to examine whether activations of a hidden layer being i.i.d.
given the assumption that the activations of the previous layer are i.i.d..
More speciﬁcally, since { F 1 n } Nn =1 are i.i.d.
, we can use the sample mean f 1 to estimate E ( F 1 ) .
Because the samples of the training dataset D are also assumed to be i.i.d.
, we can train the MLP multiple times to draw multiple independent observations of W nk and B 2 k , thus we can use the sample covariance to estimate Cov ( W k , W k (cid:48) ) ,
Cov ( W k , B 2 k (cid:48) ) , and etc.
[Table]
[Equation]
[Table]
[Equation]
Based on the empirical approach, we estimate the statistical measures (Table 1) to examine whether the activations of the three layers (i.e., f 1 , f 2 , and f Y ) being i.i.d.
in the MLP for classifying the
MNIST dataset.
Without loss of generality, we restrict all the layers of the MLP from using bias to decrease computation complexity, thus the necessary conditions for f 2 can be simpliﬁed as
[Equation]
≤ K } , we must have
[Equation]
First, Figure 3 shows the correlation matrices, i.e., A N × N , W K × K , and C L × L , to check if their corresponding covariances, i.e, Cov ( A n , A n (cid:48) ) , Cov ( W k , W k (cid:48) ) , and Cov ( C k , C k (cid:48) ) , are close to zero.
It shows that many correlations are far from zero, so their corresponding covariances are not close to zero either.
Second, we use the sample mean, e.g., x , to estimate the expectation, e.g., E ( X ) .
Taking into account the estimation error, we can regard E ( X ) =
based on x = 0 .
However, f 1 = 0 .
436 and f 2 = 0 .
498 imply E ( F 1 ) (cid:54) = 0 and E ( F 2 ) (cid:54) = 0 even considering the estimation error.
As a result, the activations of the last two hidden layer, i.e., f 2 and f Y , cannot be independent because they cannot satisfy the ﬁrst necessary condition, i.e., Equation 6.
The detailed information about the estimation of the statistical measures and the architecture of the MLP are presented in
Appendix F.1.
In addition, we show that the necessary condition for activations being identically distributed also cannot be satisﬁed by the MLP in Appendix F.1.
Furthermore, we demonstrate that the necessary conditions for activations being i.i.d.
also cannot be satisﬁed in the CNN for classifying the CIFAR-10 dataset in Appendix F.
Overall, we conclude that activations being i.i.d.
is not valid for all the hidden layers of DNNs based on the theoretical necessary conditions and the experimental simulations on typical DNNs.
In other words, we cannot use the CLT to establish the equivalence between GP and all the hidden layers of
DNNs, thus GP with the i.i.d.
prior could not correctly clarify all the hidden layers of DNNs.
That necessitates a new probabilistic representation for explaining the hidden layers of DNNs.
[Equation]
[Figure]
3. T HE DISTRIBUTION OF A HIDDEN LAYER IS A G IBBS DISTRIBUTION
This section proves that the distribution of a hidden layer can be formulated as a Gibbs distribution and the statistical connection between two adjacent layers can be modeled as a PoE model.
Assuming the output layer in the MLP (Figure 1) is the softmax, its distribution can be expressed as
[Equation]
where Z Y =
(cid:80) Ll =1 exp ( f yl ) is the partition function.
Since f yl = (cid:80) Kk =1 γ kl · f 2 k + b yl , we have
[Equation]
Based on the properties of the exponential function, i.e., exp ( a + b ) = exp ( a ) · exp ( b ) and exp ( a · b ) =
[ exp ( b )] a , we can reformulate P ( F Y ) as
[Equation]
where Z (cid:48)
Y =
Z Y / exp ( b yl ) .
Since { exp ( f 2 k ) } Kk =1 are scalar, we can introduce a new partition function Z F 2 = (cid:80)
Kk =1 exp ( f 2 k ) such that { 1 Z F 2 exp ( f 2 k ) }
Kk =1 become a probability measure.
As a result, we can further reformulate P ( F Y ) =
{ P ( f yl ) } Ll =1 as a Product of Expert (PoE) model as
[Equation]
where Z (cid:48)(cid:48) Y =
Z Y / [ exp ( b yl ) · (cid:81)
Kk =
[ Z F 2 ] γ kl ] and each expert is deﬁned as 1 Z F 2 exp ( f 2 k ) .
It is noteworthy that all the experts { 1 Z F 2 exp ( f 2 k ) } Kk =1 form a probability measure and establish an exact one-to-one correspondence to all the neurons in the second hidden layer f 2 , i.e., { f 2 l } Kk =1 .
Therefore, the distribution of f 2 can be expressed as
[Equation]
Based on the deﬁnition of Gibbs distribution, Equation 8 and 12 show that f Y and f 2 formulate two multivariate Gibbs distributions and their energy functions are equivalent to the negative of the output nodes { f yl } Ll =1 and the neurons { f 2 k } Kk =1 , respectively.
In addition, Equation 11 implies that the connection between P ( f yl ) and P ( f 2 k ) can be expressed as the PoE model, namely that P ( f yl ) is equal to the product of all the experts { P ( f 2 k ) }
Kk =1 with their respective weights { γ kl } Kk =1 .
Compared to the GP explanations assuming all the activations { f 2 k } Kk =1 are i.i.d.
and deriving
P ( f yl ) being a Gaussian distribution in the limit of inﬁnite neurons, i.e., { f 2 k } K →∞ k
=1 , based on CLT, we formulate P ( f yl ) as a PoE model, in which all the experts { 1 Z F 2 exp ( f 2 k ) }
Kk =1 are deﬁned by all the neurons { f 2 k } Kk =1 .
Overall, the above probabilistic representation provides a more speciﬁc explanation for the hidden layer without any assumption.
Moreover, we prove that the probabilistic representation is valid for other hidden layers in the MLP.
Since { f 2 k = σ 2 ( (cid:80) Nn =1 β nk · f 1 n + b 2 k ) }
Kk =1 , P ( F 2 ) can be expressed as
[Equation]
Based on the equivalence between the gradient descent algorithm and the ﬁrst order approximation (Battiti, 1992), we prove that σ 2 ( (cid:80) Nn =1 β nk · f 1 n + b 2 k )] can be approximated as
[Equation]
where C 21 and C 22 only depend on the activations { f 1 n } Nn =1 in the previous training iteration, thus they can be regarded as constants and absorbed by β nk and b 2 k .
The proof for the approximation is included in Appendix G. Therefore, P ( F 2 ) also can be modeled as a PoE model.
[Equation]
where Z
[Equation]
=
Z
[Equation]
[ exp
[Equation]
=
[Equation]
[ Z
[Equation] [Equation]
nk ] and Z F 1 = (cid:80) Nn =1 exp ( f 1 n ) .
Finally, the distribution of the ﬁrst hidden layer f 1 is formulated as
[Equation]
Based on the same derivation as P ( F 2 ) , P ( F 1 ) also can be modeled as a PoE model as follows.
[Equation]
where Z (cid:48)(cid:48) F 1 = Z F 1 / [ exp ( b 1 n ) · (cid:81)
Mm =
[ Z X ] α mn ] and Z X
= (cid:80)
Mm =1 exp ( x m ) .
In summary, we introduce a novel probabilistic representation for the hidden layers in two aspects.
First, a fully connected layer formulate a multivariate Gibbs distribution (Equation 8, Equation 12, and Equation 16) to describe the probability of the occurrence of multiple features formulated by their respective neurons and their energy functions are equivalent to the negative of their respective neurons.
Second, the statistical connection between two adjacent fully connected layers can be expressed by a PoE model (Equation 11, Equation 15, and Equation 17).
Furthermore, we prove that a convolutional layer formulates a speciﬁc Gibbs distribution, i.e., the
MRF model, in Appendix H. Speciﬁcally, a convolutional layer
f i with K convolutional channels, i.e., f i = { f ki } Kk =1 , in a DNN = { x ; f 1 ; ... ; f I ; f Y } can be formulated as a MRF model.
[Equation]
where f ki = (cid:80) Nn =1 σ i ( S ( k,n ) i ◦ f ni − 1 )+ b ki · 1 is the k th convolutional channel derived from the previous layer with N channels, i.e., f i − 1 =
{ f ni − 1 } Nn =1 , S ( k,n ) i is the convolutional ﬁlter connecting f ki and f ni − 1 , b ki is the bias, and σ i ( · ) is the activation function.
In particular, its energy function is equal to the negative summation of all the convolutional channels, i.e., E F i = − (cid:80)
Kk =
1 f ki
S IMULATIONS
This section uses two synthetic datasets to demonstrate the proposed probabilistic representation: (i) the distribution of a fully connected layer is a multivariate discrete Gibbs distribution and (ii) the distribution a convolutional layer is a MRF model.
4. T HE DISTRIBUTION OF A FULLY CONNECTED LAYER IS A MULTIVARIATE DISCRETE G IBBS DISTRIBUTION
A fully connected layer f
= { f n } Nn =1 with N neurons formulates a multivariate Gibbs distribution to describe the probability of the N features deﬁned the neurons occurring in the input.
Their energy functions are equal to the negative of their respective neurons, i.e., E f n = − f n ( x ) .
[Equation]
(19) where f n ( x ) =
σ [ g n ( x )] , g n ( x )= (cid:80)
Mm =1 α mn · x m + b n is a linear ﬁlter with α mn being the weights and b n being the bias, σ ( · ) is the activation function, and Z F = (cid:80) N n =1 exp
[ f n ( x )] is the partition function.
Since benchmark datasets consist of very complex features and typical DNNs have too complicated architecture, it is very hard to directly derive the statistical connection between the training dataset and the learned parameters of DNNs.
Alternatively, we choose the neural network for classifying the synthetic dataset in Appendix A.4 to demonstrate the probabilistic representation, because the neural network only has a single fully connected hidden layer with seven neurons, i.e., N = 7 , and the synthetic dataset only has four features.
After the network is trained well, we choose a synthetic image as the input and show all the information of the learned hidden layer in Figure 4.
Since the synthetic dataset contains four different features in the primary and secondary diagonal directions, the neurons are shown to learn seven ﬁlters to distinguish these features.
Also note that the input image is sorted in the primary diagonal direction, i.e., it has a huge difference between the two sides of the secondary diagonal (the red curve), thus Neuron1, Neuron 2 and Neuron 4 derive large positive output (i.e. g 1 ( x ) = 122 . 09 , g 2 ( x ) = 113 .
70 , and g 4 ( x )
= 140 . 20 ), but Neuron6 and
Neuron 7 derive large negative output (i.e. g 6 ( x ) = − 134 .
08 , g 7 ( x ) = − 133 . 36 ).
After applying the sigmoid function, we have f 1 ( x ) =
f 2 ( x ) =
f 4 ( x ) = 1 and f 6 ( x )
=
f 7 ( x ) = 0 .
Based on the deﬁnition of Gibbs distribution (Equation 19), the Gibbs probability of different neurons are equal to P ( f 1 ) =
P ( f 2 ) = P ( f 4 ) = 0 .
20 and P ( f 6 ) =
P ( f 7 ) = 0 . 07 , which indicates that the probability of the occurrence of the features deﬁned by Neuron1, Neuron2 and Neuron4 is high, but the probability corresponding to Neuron6 and Neuron 7 are very low in the synthetic image.
4. T HE DISTRIBUTION OF A CONVOLUTIONAL LAYER IS A MRF MODEL
Since the distributions of benchmark datasets are unknown, it is impossible to use them to verify the probabilistic explanation for convolutional layers.
Alternatively, we generate a synthetic dataset obeying the Gaussian distribution N (0 , 1024) based on the NIST dataset of handwritten digits.
It consists of 20,000 32 × 32 grayscale images in 10 classes (digits from 0 to 9), and each class has 1,000 training images and 1,000 testing images.
Figure 5 shows ﬁve synthetic images and their perspective histograms.
The method for generating the synthetic dataset is included in Appendix I.
We design a CNN = { x ; f 1 ; f 2 ; f 3 ; f 4 ; f Y } for classifying the synthetic dataset.
The CNN has two convolutional layers with ReLU (i.e., f 1 and f 3 ), two max-pooling layers (i.e., f 2 and f 4 ) and a softmax output layer f Y .
The detailed information about the CNN is included in Appendix I.
Based on the MRF explanation (Equation 18), the distribution of f 1 can be formulated as
[Equation]
In particular, the energy function of P ( F 1 ) is equivalent to E F 1 = − (cid:80) 20 k =1 f k 1 .
[Figure]
[Figure] [Figure]
[Figure] [Figure]
In order to model a high dimensional dataset, e.g., the 32 × 32 synthetic image
[Figure]
we typically need a multivariate distribution with the same dimension
[Figure]
e.g., the dimension of the covariance function of a Gaussian process should be 1024 × 1024 .
However, most structured probabilistic graphical models, especially the MRF model, have two fundamental assumptions: stationary and Markov (Lyu & Simoncelli, 2007).
The former means that the distribution is independent on the location,
i
[Figure]
e., the distribution of pixels at different locations are identically distributed.
The later indicates that the distribution of a single pixel is independent on the other pixels given its neighbors.
The two assumptions allow us to use the histogram of samples to simulate the true distribution
[Figure]
( X ) || P ( F 1 )]
= 0 .
After the CNN is well trained, we randomly choose a testing image x as the input to derive P ( F 1 ) .
Since x ∼ P ( X ) , we can use the histogram of the testing image, i.e., P ( x ) , to estimate P ( X ) .
In addition, E F 1 is a sufﬁcient statistics for P ( F 1 ) that allows us to use the histogram of E F 1 to estimate P ( F 1 ) .
As a result, we can verify the probabilistic representation by calculating the distance between P ( F 1 ) and P ( X ) , i..
e, KL [ P ( X ) || P ( F 1 )] .
We visualize their distributions in Figure 6, which shows that P ( F 1 ) is very close to P ( X ) , i.e., KL ( P ( X ) ||
q ( F 1 )) = 0 .
Therefore, the distribution of a convolutional layer can be modeled as a MRF model.
In addition, this experiment provides a viewpoint to show the activations of different convolutional layers not being i.i.d..
Given another synthetic image, we visualize the output of ﬁve convolutional channels in f 1 in Figure 7.
It shows that the activations of different channels at the same position are correlated, i.e., they cannot be independent.
In addition, we estimate the distribution of each convolutional channel based on the histograms of their respective energy functions.
It shows that the distributions of different channels are different, i.e., they cannot be identically distributed.
[Figure]
D ISCUSSION 5. T HE ENTIRE ARCHITECTURE OF DNN S CAN BE EXPLAINED AS A BHM
Since the output of the hidden layer f i in a DNN = { x ; f 1 ; ... ; f I ; f Y } is commonly the input of the next layer
f i +1 , the DNN forms a Markov chain as F 1 → · · · → F I → F Y .
Hence, the distribution of a hidden layer in DNNs should be viewed as a conditional distribution.
As a result, the entire architecture of DNNs formulates a joint distribution as
[Equation]
Since P ( F Y | F I ) is an estimation of the true distribution P ( Y | X ) , it can be regarded as a likelihood distribution, and the simulations show that P ( F 1 | X ) can be regarded as a prior distribution of the training dataset.
The Markov chain indicates that other hidden layers recursively serve as prior distributions for their respective next layers, i.e., P ( F i +
| F i ) .
Therefore, the entire architecture of DNNs can be explained as a BHM, which provides a novel probabilistic explanation for the hierarchy property of DNNs, which cannot be clariﬁed by GP (Matthews et al., 2018)
5. A NOVEL REGULARIZATION ALGORITHM FOR DNN S
In contrast to recent works claim that DNNs perform implicit regularization by the backpopagation learning (Zhang et al., 2016; Neyshabur et al., 2017), the BHM explanation indicates that DNNs have explicit regularizations represented by the hidden layers serving as prior distributions based on the
Bayesian regularization theory (Steck & Jaakkola, 2003).
This novel explanation for DNNs indicates a new regularization approach to improve the generalization performance of DNNs through pretraining the hidden layers corresponding to prior distributions.
In particular, we introduce a novel regularization algorithm for CNNs and demonstrate its superiority over traditional regularizations, e.g., dropout (Srivastava et al., 2014), based on the CIFAR-10 classiﬁcation task in Appendix J.
C ONCLUSION
In this paper, we demonstrate that the parameters of DNN cannot satisfy the i.i.d assumption and the activations being i.i.d.
cannot be valid for all the hidden layers of DNNs.
Hence, GP cannot correctly explain all the hidden layer of DNNs.
Alternatively, we propose a novel probabilistic representation for DNNs in tree aspects: (i) a hidden layer formulates a Gibbs distribution, in which neurons deﬁne the energy function, (ii) the connection between two adjacent layers can be modeled by a PoE model, and (iii) the entire architecture of DNNs can be explained as a BHM.
There are two general directions for future research.
The theoretical direction is to examine the probabilistic representation in other popular DNNs, e.g., the residual networks, which can further verify the proposed probabilistic representation.
The practical direction is to validate the proposed regularization in more complex DNNs and datasets.
A T HE PARAMETERS OF DNN S CANNOT SATISFY THE I . I . D . PRIOR A. A N EURAL N ETWORK WITH A SINGLE HIDDEN LAYER ON THE MNIST DATASET
Based on the seminal work (Neal, 1994), we design a neural network with a single fully connected hidden layer (Figure 8) for classifying the benchmark MNIST dataset (LeCun et al., 1998).
The input layer dimension is 1 × 784 , the hidden layer has 128 neurons with the sigmoid activation function, and the output layer is the softmax with 1 × 10 dimension.
Table 2 summarizes the architecture of the network.
In addition, { α mn } Mm =1 denote the weights of the n th linear ﬁlter in the fully conntected hidden layer and { β nl } Nn =1 are the weights of the l th linear ﬁlter in the output layer.
Therefore, the hidden layer has 128 linear ﬁlters with 1 × 784 dimension, i.e., { α mn } 784 m =1 .
The output layer has totally 10 linear ﬁlters and their dimension is 1 × 128 ,
i.e., { β nl } 128 n =1 .
[Figure]
⋮
[Figure]
After the network is well trained (its training and testing accuracy is 97 .
4% and 96 .
9% , respectively), we calculate the absolute sample correlations, i.e., | r ( α n , α n (cid:48) )
| and | r ( β k , β k (cid:48) ) | , and derive two absolute correlation matrices, i.e., A (cid:48)
N × N and B (cid:48)
L × L , based on Equation 2.
They are shown in
Figure 9.
We ﬁnd that most | r ( α n , α n (cid:48) ) | are close to zero, but | r ( β k , β k (cid:48) )
| not.
That indicates the parameters of the NN do not satisfy the necessary condition, thus they cannot satisfy the i.i.d.
prior.
[Figure]
[Table]
A. A MLP ON THE MNIST DATASET
We design a MLP (Figure 1) to classify the benchmark MNIST dataset.
The MLP has two hidden layers, each hidden layer is a fully connected layer with the sigmoid activation function, and the output layer is the softmax.
Table 3 summarizes the architecture of the MLP.
After the MLP is well trained (its training and testing accuracy is 97 .
7% and 97 .
1% , respectively), we quantify the correlation of different weights in the same layer based on Equation 22 and derive the correlation matrix for all the weights in every hidden layer.
[Equation]
where α n = 1 M (cid:80) Mm =1 α mn is the sample mean of { α mn } Mm =1 .
To examine if different weights are correlated, we only need to check whether their correlation coefﬁcients are close to zero, so we take the absolute value of the correlation matrix element-wisely.
Figure 10 visualizes the correlation matrices of each hidden layer, i.e., A (cid:48)
N × N , B (cid:48)
K × K , and C (cid:48)
L × L .
We ﬁnd that most weights in f 1 are uncorrelated because their correlation coefﬁcients are very close to zero, i.e., | r ( α n , α n (cid:48) )
| ≈ 0 , but many weights in f 2 and f Y are correlated to others, i.e., | r ( β k , β k (cid:48) )
| (cid:54) = 0 and | r ( γ l , γ l (cid:48) )
| (cid:54) = 0 .
In addition, we derive the average of the absolute correlation coefﬁcients of all the weights in each layer and show them at the top of Figure 10, which further validates that the weights of some hidden layer are correlated.
Therefore, the parameters of the MLP cannot satisfy the i.i.d.
prior.
[Figure]
[Table]
A. A CNN ON THE CIFAR-10 DATASET
We design a CNN to classify the benchmark CIFAR-10 dataset (Krizhevsky, 2009).
The CNN has two convolutional layers without activation function, two max-pooling layers for dimension reduction, and two fully connected layers with the sigmoid activation function.
In addition, the output layer is the softmax.
Table 4 summarizes the architecture of the CNN.
After we train the CNN well (its training and testing accuracy are 98 .
0% and 69 .
4% , respectively).
We quantify the correlation of different weights in the same layer based on Equation 2, thereby deriving the absolute correlation matrix for all the weights in every hidden layer.
Since x has 3 channels and f 1 has 64 channels, f 1 has totally 3 × 64 = 192 ﬁlters.
Hence, the dimension of the correlation matrix for f 1 is 192 × 192 .
In particular, we ﬂatten the output of the second convolutional layer, i.e., reshaping 8 × 8 × 128 to 1 × 8192 , thus the ﬁlter dimension of f 3 is 8192 × 1024 .
Table 4 shows the dimension of the correlation matrices for all the layers in the CNN.
Figure 11 shows the absolute correlation matrices for every layer.
We can ﬁnd that most weights are correlated in f 1 and f 2 , i.e., the weights in the convolutional layers are correlated.
In addition, the weighs in the output layer are also shown to be correlated.
However, the weights in f 3 and f 4 are not correlated, because their correlation coefﬁcients are close to zero.
In other words, the weights in the fully connected layers are uncorrelated.
Moreover, we derive the average of the absolute correlation coefﬁcients of all the weights in each layer and show them at the top of Figure 11, which further validates the correlation of the weights in each layer.
Overall, the parameters of the CNN cannot satisfy the necessary condition.
Therefore, they cannot satisfy the i.i.d.
prior.
[Figure]
A. N EURAL NETWORKS WITH A SINGLE HIDDEN LAYER ON A SYNTHETIC DATASET
In order to further demonstrate that the parameters of DNNs cannot satisfy the i.i.d.
assumption, we construct a neural network, i.e., NN = { x ; f ; f Y } shown in Figure 8, to classify a synthetic dataset.
The synthetic dataset consists of 4,800 32 × 32 grayscale images with 4 classes, and every synthetic image is sampled from the Gaussian distribution N (0 , 1) .
To make the synthetic image have certain features that can be learned by neural networks, we sort the synthetic image in the primary or secondary diagonal directions by the ascending or descending orders.
Thus, the synthetic dataset only has four features, i.e., four classes, and their corresponding images are drawn above.
There are two important reasons why we choose the synthetic dataset.
Above all, compared to benchmark datasets containing too complex features, the synthetic dataset only has four simple features, thus we can easily show the correlation of different parameters by simply visualizing them.
In addition, since a single synthetic image only has a single feature, we can easily examine the connection between the parameters of networks and the synthetic image.
[Figure]
Since the synthetic dataset is very simple, we set the hidden layer f have only eights neurons and the output layer f Y have four output nodes.
After training the network well, we show all the eight neurons in Figure 13.
In contrast to the i.i.d.
assumption, their weights show strong correlation in three aspects.
First, weights show strong internal correlation within each neuron, i.e., if a weight has large magnitude, its neighbors have high probability to be large, and vice versa.
Second, weights show strong external correlation between different neurons.
For example, most weights of Neuron and Neuron5 at the same position show strong positive correlation, and most weights of Neuron and Neuron7 at the same position show strong negative correlation.
Third, the weights show strong dependence on the training dataset.
More speciﬁcally, since all the features of the synthetic dataset concentrate in the primary and secondary diagonal directions, most linear ﬁlters also demonstrate related features in the same directions.
[Figure] [Figure]
[Figure] [Figure]
Figure 14 visualizes the four linear ﬁlters in the output layer.
Though they not show too strong internal correlation because their dimension is too small, their weights at the same position still show strong external correlation, e.g., the weights of Filter0 and
Filter1 at the same position show strong negative correlation.
Moreover, we calculate the correlation coefﬁcients of all the four ﬁlters based on Equation 2 and derive the correlation matrix show in Figure 14.
It shows that the correlation coefﬁcient of Filter0 and Filter1 is close to − 1 , which further validates that they have strong negative correlation, hence the parameters of the output layer do not satisfy the i.i.d.
prior.
Compared to the simulations using benchmark datasets, the synthetic dataset enable us to expose the correlation of different weights and their dependence on the training dataset in the more clearly way.
Overall, we demonstrate that the parameters of neural networks are not i.i.d..
B P ROOF : T HE NECESSARY CONDITION FOR THE ACTIVATIONS OF HIDDEN LAYERS BEING INDEPENDENT
[Equation]
[Figure]
σ 1 [ g 1 n ( x )] } Nn =1 , where g 1 n ( x ) = (cid:80)
Mm =1 α mn · x m + b 1 n is the n th linear function with α mn being the weight of the edge between x m and f 1 n and b 1 n being the bias, and σ 1 is a non-linear activation function.
In addition, f 2 = { σ 2 [ g 2 k ( f 1 )] } Kk =1 where g 2 k ( f 1 ) =
(cid:80) Nn =1 β nk · f 1 n + b 2 k and f Y
= { σ 3 [ g yl ( f 2 )] } Ll =1 where g yl ( f 2 ) =
(cid:80)
Kk =1 γ kl · f 2 k + b yl .
They have the similar deﬁnitions as f 1 n .
In the context of Bayesian probability, all the weights and the biases in the MLP (Figure 15) have prior distributions, i.e., α mn ∼ P ( A mn ) , β nk ∼ P
( W nk ) , b 1 n ∼ P
( B 1 n ) , and b 2 k ∼ P ( B 2 k ) , thus we regard G 1 n = (cid:80)
Mm =
A mn X m + B 1 n as the random variable of g 1 n ( x ) = (cid:80)
Mm =1 α mn · x m + b 1 n , and G 2 k
= (cid:80) Nn
=1 W nk F 1 n + B 2 k as the random variable of g 2 k ( f 1 ) =
(cid:80) Nn =1 β nk · f 1 n + b 2 k .
Therefore, the random variables of arbitrary activations in f 1 and f 2 , e.g., f 1 n and f 2 k , can be expressed as F 1 n = σ 1 ( G 1 n ) and F 2 k = σ 2 ( G 2 k ) , respectively.
Based on the theorem (Appendix D) that functions of independent random variables are independent, we can derive that if { F 2 k } Kk =1 are independent and the activation function σ 1 ( · ) is invertible, then { G 2 k } Kk =1 are independent.
In other words, in order to prove { F 2 k } Kk =1 are independent, we only need to prove { G 2 k } Kk =1 being independent as long as σ 2 ( · ) is invertible.
The necessary condition for { G 2 k } Kk =1 being independent is ∀ ( k, k (cid:48) )
∈ S 1 = { ( k, k
[Equation]
) ∈
Z 2 |
[Equation]
= k (cid:48) , 1 ≤ k ≤ K, 1 ≤ k (cid:48) ≤ K } , the covariance Cov ( G 2 k , G 2 k (cid:48) )
= 0 , which can be formulated as
[Equation]
Based on Cov ( X + Y, W + Z
[Equation]
=
[Equation]
, we can derive that
[Equation] [Equation]
Based on the deﬁnition of covariance
[Equation]
i
[Equation]
e., Cov ( A, B ) =
E ( AB ) − E
( A ) E ( B ) , we have
[Equation]
Based on the law of total expectation
[Equation]
i.e, E ( AB ) =
E A ( AE B | A ( B )) , we have
[Equation]
[Equation]
Since the statistical properties of the input { F 1 n } Nn =1 are unrelated to the weights β nk and biases b 2 k , we have E F 1 n F 1 n
[Equation]
W nk W
[Equation]
[Equation] [Equation]
1 n )
[Equation]
E F 1 n ( F 1 n ) , and E F 1 n | W nk ( F 1 n ) =
E F 1 n ( F 1 n ) .
As a result
[Equation]
we can derive that
[Equation]
Substituting Equation (27) for the corresponding expectations in Equation (25)
[Equation]
we have
[Equation]
Assuming { F 1 n } Nn =1 are i.i.d.
and F 1 n ∼ P ( F 1 ) , we have E ( F 1 n F 1 n (cid:48) )
=
E ( F 1 n ) E ( F 1 n (cid:48) )
=
E 2 ( F 1 ) .
Hence, we can derive that
[Equation]
Substituting Equation (29) for the corresponding covariances in Equation (24), we have
[Equation]
Based on Cov ( X + Y, W + Z ) =
Cov ( X + W ) + Cov ( X
[Equation]
Z )
[Equation]
Cov
[Equation]
Y
[Equation] [Equation]
Y + Z ) , we can derive that
[Equation]
Therefore, given σ 2 ( · ) is invertible and { F 1 n } Nn =1 are i.i.d., the necessary condition for { F 2 k } Kk = being independent can be expressed as
[Equation]
where W k = (cid:80)
[Equation]
and W
[Equation]
Overall, in order to guarantee the activations of a hidden layer are independent with each other in the context of Bayesian probability, the prior distributions of the weights and biases of the hidden layer should satisfy the above necessary condition given the inputs of the hidden layer are i.i.d.
and the activation functions are invertible.
C P ROOF : T HE NECESSARY CONDITION FOR THE ACTIVATIONS OF HIDDEN LAYERS BEING IDENTICALLY DISTRIBUTED
Given the same DNN drawn in Figure 15, we assume σ 2 ( · ) being strictly increasing and differentiable, so σ 2 ( · ) is invertible and its inverse σ − 1 2 ( · ) is also strictly increasing.
The cumulative distribution function of F 2 k can be expressed as
[Equation]
) where φ ( F 2 k ≤ f ) is the probability of F 2 k takes on a value less than or equal to f .
Subsequently, we can obtain that
[Equation]
Equation (34) means that if the activation function σ 2 ( · ) is strictly increasing and differentiable, { F 2 k } Kk =1 being identically distributed implies { G 2 k } Kk =1 being identically distributed.
In the context of Bayesian probability, G 2 k = (cid:80)
Nn =1 W nk F 1 n + B 2 k , we can derive that
[Equation]
Based on the law of total expectation and Equation 27, we have
[Equation]
Assuming { F 1 n } Nn =1 are i.i.d.
and F 1 n ∼ F 1 , we have E ( F 1 n ) =
E ( F 1 ) .
Hence, we can derive that
[Equation]
Based on the property of expectation, i.e., E ( A ) + E ( B ) =
E ( A + B ) , we have
[Equation]
Therefore, given σ 2 ( · ) is strictly increasing and differentiable and { F 1 n } Nn =1 are i.i.d., the necessary condition for { F 2 k } Kk =1 being identically distributed is that ∀ ( n, n (cid:48) )
∈ S 1 , we have
[Equation]
where W k = (cid:80) Nn =1 W nk and W k (cid:48)
= (cid:80) Nn =1 W nk (cid:48) .
Overall, in order to guarantee the activations of a hidden layer are identically distributed, the prior distributions of the weights and the biases of the hidden layer should satisfy the above necessary condition in the context of Bayesian probability given the inputs of the hidden layer are i.i.d.
and the activation functions are strictly increasing and differentiable.
D T HEOREM : F UNCTIONS OF INDEPENDENT RANDOM VARIABLES ARE INDEPENDENT
Let X and Y be independent random variables on a probability space (Ω , F , P ) .
Let g and h be real-valued functions deﬁned on the codomains of X and Y , respectively.
Then g ( X ) and h ( Y ) are independent random variables.
Proof: Let A ⊆ R and B ⊆ R be the range of g and h , the joint distribution between g ( X ) and h ( Y ) can be formulate as P ( g ( X )
∈
A , h ( Y ) ∈ B ) .
Let g − 1 ( A ) and h − 1 ( B ) denote the preimages of A and B , respectively, we have
[Equation]
[Equation]
Based on the deﬁnition of independence, we can derive that
[Equation]
Based on the deﬁnition of preimage, we can derive that
[Equation]
Therefore, g ( X ) and h ( Y ) are independent random variables.
E T HE NECESSARY CONDITIONS FOR ACTIVATIONS BEING I . I . D . ARE VALID FOR CONVOLUTIONAL LAYERS
Assuming a DNN = { x ; f 1 ; ...
; f I ; f Y } has two adjacent convolutional layers f i and f i +1 .
The ﬁrst convolutional layer has N channels, i.e. f i = { f ni } Nn =1 , and the second convolutional layer has
K channels, i.e. f i +1 =
{ f ki +1 } Kk =1 .
Therefore, the dimension of the convolutional ﬁlters
S i +1 in f i +
are W × H × N × K , i.e., the dimension of a single ﬁlter connecting the f ni channel and the f k i +1 channel are D ( S ( k,n ) i +1 ) =
W × H .
As a result, the f k i +1 channel can be formulated as
[Equation]
(43) where σ i +1 is the activation function and b i +1 is the bias for the ( i + 1) th channel.
A B
C D 𝑥 " 𝑥 # 𝑥 $ 𝑥 % 𝑥 & 𝑥 ’ 𝑥 ( 𝑥 ) 𝑥 *
A B
C D
A B
C D
A B
C D
A B
C D 𝑥 " 𝑥 # ⋮ 𝑥 ) 𝑥 * ∘ 𝑦 "
𝑦 # 𝑦 $ 𝑦 %
A convolution channel
A fully connected layer
Convolutional filters
Input
Output 𝑦 " =
𝐴 0 𝑥 " + 𝐵 0 𝑥
# + 𝐶
𝑥 % + 𝐷 0 𝑥 & 𝑦 # =
𝐴 0 𝑥
+ 𝐵 0 𝑥
$ + 𝐶 0 𝑥 & + 𝐷 0 𝑥 ’ 𝑦 $ =
𝐴
𝑥 %
+ 𝐵 0 𝑥 & + 𝐶 0 𝑥 ( + 𝐷 0 𝑥 ) 𝑦 %
=
𝐴 0 𝑥 & + 𝐵 0 𝑥 ’ + 𝐶 0 𝑥 )
+ 𝐷 0 𝑥 * = 𝑏 𝑥6 " 𝑥6 # 𝑥6 $ 𝑥6 % 𝑥6 & 𝑥6 ’ 𝑥6 ( 𝑥6 ) 𝑥6 *
E F
G H ∘ 𝑦6 " 𝑦6 # 𝑦6 $ 𝑦6 % = 𝑧 " 𝑧 # 𝑧 $ 𝑧 %
+ +
E F
G H
E F
G H
E F
G H
E F
G H 𝑥6 " 𝑥6 # ⋮ 𝑥6 ) 𝑥6 * 𝑏 + = 𝑧 " 𝑧 # 𝑧 $ 𝑧 % =
[Figure]
is the
[Figure]
𝒙
Since Garriga-Alonso et al.
(2018) demonstrate the equivalence between a convolution operation and the matrix multiplication, we can regard the convolutional channel f ki +1 as a fully connected layer.
Speciﬁcally, the input of the virtual fully connected layer is the same as the input of the convolutional channel, the output of the virtual fully connected layer is the vectorized output of the convolutional channel, and the weights of the virtual fully connected layer depend on the convolutional ﬁlter and the spatial location of the output of the convolutional channel.
Without considering the activation function, the equivalence between a convolutional channel and a fully connected layer can be explained in Figure 16, in which the ﬁrst convolutional layer has two channels, i.e., f i = { x ; ˆ x } , where x = { x 1 , · · · , x 9 } and ˆ x = { ˆ x 1 , · · · , ˆ x 9 } , and the k th channel of the second convolutional layer is f ki +
=
{ z 1 , z 2 , z 3 , z 4 } .
In addition, the dimension of the convolutional ﬁlter S i +1 is 2 × 2 × 2 × 1 , where S ( k, 1) i +
=
[ A, B ; C, D ] and S ( k, 2)
i +1 =
[ E, F ; G, H ] .
Figure 16 shows that the output of f ki +1 can be expressed as a fully connected layer.
The input of the fully connected layer is the vector x , the weights of each neuron is represented by each row of the matrix W , and the bias is b , thus f k i +
= W · x + b · 1 .
If we consider the activation function and make a precise statement, f ki +1 can be formulated as
[Equation]
(44) where W ki +1 is the matrix corresponding to all the convolutional ﬁlters in the k th channel.
To derive the necessary conditions for two arbitrary different channels are i.i.d.
, we need to derive the necessary conditions for two arbitrary activations in two arbitrary different channels are i.i.d.
For example, if the m th activation of the k channel and the m (cid:48) th activation of the k (cid:48) channel in f i +1 are i.i.d., the necessary conditions are Cov ( f ki +1 ( m ) , f k (cid:48)
i +1 ( m (cid:48) ))
= 0 , E ( f ki +1 ( m ))
=
E ( f k (cid:48)
i +1 ( m (cid:48) )) .
Since we demonstrate the equivalence between a convolutional channel and a fully connected layer, the necessary conditions for activations being i.i.d.
in a fully connected layer are also valid for a convolutional layer.
In other words, we can also use the Equation 3, Equation 4, and Equation 5 to examine whether activations being i.i.d.
is valid in a convolutional layer under the assumption that all activations being i.i.d.
in the previous layer.
[Table]
F A CTIVATIONS BEING I . I . D . CANNOT BE VALID FOR ALL THE HIDDEN LAYERS OF DNN S
In this section, we demonstrate that activations being i.i.d.
cannot be valid for all the hidden layers of DNNs through showing that the hidden layers of two typical DNNs, i.e, the MLP and the CNN, cannot satisfy the necessary conditions for activations being i.i.d..
F. T HE MLP ON THE MNIST DATASET
The architecture of the MLP for classifying the MNIST dataset is visualized in Figure 15, and its speciﬁc parameters are summarized in Table 3.
Since the activation functions of all the hidden layers in the MLP are the sigmoid function, which is strictly increasing, differentiable, and invertible, we only need to examine the ﬁrst two necessary conditions in the MLP.
Since we restrict all the hidden layers from using the bias, the ﬁrst two necessary conditions for activations being i.i.d.
can be simpliﬁed as
[Equation]
2 | k
[Equation]
=
[Equation]
1 ≤
[Equation]
≤ K
[Equation]
≤ k
[Equation]
≤ K } , we must have
[Equation]
Based on the architecture of the MLP, Table 5 summarizes all the statistical measures for each layer of the MLP.
Therefore, the key to examine if the layers of the MLP satisfy the necessary conditions is to estimate the statistical measures.
To estimate the expectations E ( X ) , E ( F 1 ) , and E ( F 2 ) , we use the sample means x , f 1 , and f 2 , respectively.
The deﬁnition of the sample mean is
[Equation]
After the MLP is well-trained, we derive the sample means as x = 0 .
131 , f 1 = 0 .
436 , and f 2 = 0 .
498 given the benchmark MNIST testing dataset.
Taking into account the estimation error, we can regard E ( X )
= 0 .
As a result, we can derive that ∀ ( n, n (cid:48) ) ,
Cov ( G 1 n , G 1 n (cid:48) )
=
and E ( G 1 n ) =
E ( G 1 n (cid:48) )
= 0 , thus f 1 satisfy the necessary conditions for activations being i.i.d..
However, f 1 = 0 .
436 and f 2 = 0 .
498 imply that E ( F 1 ) (cid:54) = 0 and E ( F 2 ) (cid:54) = 0 even considering the estimation error.
Therefore, we need to estimate Cov ( W k , W k (cid:48) ) and E ( W k ) to examine whether the activations { f 2 k } Kk =1 being i.i.d..
Since the training dataset is commonly viewed as i.i.d., we can use the same training dataset to train the MLP several times to derive multiple independent observation samples of { β nk ( t ) }
Tt =1 to estimate the random variable W nk , thereby estimating Cov ( W k , W k (cid:48) ) and E ( W k ) .
[Figure]
n
E ( A n ) f 1 (6.75)   60 k 1.
E ( W k ) f 2 (0.41)    8 l
E ( C l ) f Y (4.06)
More speciﬁcally, we use the sample correlation r ( β k , β k (cid:48) ) to check whether Cov ( W k , W k (cid:48) ) are close to zero and use the sample mean β k to estimate E ( W k ) .
[Equation]
where β k = 1 T (cid:80)
Tt =
1 β k ( t ) is the sample mean of { β k ( t ) }
Tt =1 and β k ( t ) =
(cid:80) Nn =1 β
nk ( t ) .
In order to make a tradeoff between the estimation precision and computation complexity, we use the same training dataset to train the MLP 20 times to derive the samples, i.e., { β nk ( t ) } 20 t =1 .
For an activation f 2 k , it has 128 weights because its previous layer has 128 neurons.
Hence, a single sample β k ( t )
=
(cid:80) 128 n =1 β nk ( t ) .
Based on the 20 samples, i.e., { β k ( t ) } 20 t =1 , we can calculate r ( β k , β k (cid:48) ) to examine whether Cov ( W k , W k (cid:48) ) are close to zero and derive the sample mean to estimate E ( W k ) .
We use the same method to estimate Cov ( A n , A n (cid:48) ) , E ( A n ) , Cov ( C l , C l (cid:48) ) , and E ( C l ) .
Finally, we shows the absolute correlation matrices i.e., A N × N , W K × K , and C L × L , in Figure 3.
Since many elements in the three matrices are far from zero, we conclude that their corresponding covariances, i.e, Cov ( A n , A n (cid:48) ) ,
Cov ( W k , W k (cid:48) ) , and Cov ( C k , C k (cid:48) ) , are not close to zero based on connection between correlation and covariance
[Equation]
where σ W k > 0 is the standard derivation of W k .
Figure 17 visualizes the expectations, i.e., E ( A n ) , E ( W k ) , and E ( C l ) , for each layer in the MLP.
We ﬁnd that some expectations show great variations, especially { E ( A n ) } Nn =1 and { E ( C l ) } Ll =1 .
In other words, the MLP cannot guarantee that all the expectations of the weights summation are equivalent even though we take into account the estimation error.
Therefore, the MLP cannot satisfy the necessary condition for activations being identically distributed.
Overall, we conclude that not all the hidden layers of the MLP can satisfy the necessary conditions for activations being i.i.d.
under the assumption that the activations of the previous layer are i.i.d.
F.
T HE CNN ON THE CIFAR-10 DATASET
The architecture of the CNN for classifying the CIFAR-10 dataset is in Table 4.
In particular, the two convolutional layers do not have activations function, i.e., their activation functions can be viewed as y = x , which is strictly increasing, differentiable, and invertible.
The sigmoid activation function is also strictly increasing, differentiable, and invertible, thus we only need to examine the ﬁrst two necessary conditions in the CNN.
To simplify derivation and decrease computation complexity, we assume the max-pooling layers are special activation functions and satisfy the third necessary condition for activations being i.i.d.
[Table]
Similar as the MLP, we also restrict all the layers of the CNN from using the bias, thus the ﬁrst two necessary conditions for activations being i.i.d.
can be simpliﬁed as
[Equation]
} , we must have
[Equation]
Table 6 summarizes all the statistical measures for every layer of the CNN based on the architecture of the CNN in Table 4.
The deﬁnitions of most random variables are similar as the MLP except the dimension of the weights.
To check if two arbitrary convolutional channels are i.i.d.
, we need to know all the weights of two convolutional channels.
Based on the equivalence between a convolutional channel and a fully connected layer (Appendix E), all the weights of a single convolutional channel consist of all the convolutional ﬁlters related to the convolutional channel.
For example, the dimension of all the convolutional ﬁlters related to a convolutional channel in f 1 are 3 × 3 × 3 , thus a single convolutional channel in f 1 has 27 weights, so the weights dimension of f 1 is 27 × 64 .
After training the CNN well (its training accuracy is 96 . 2% and testing accuracy is 67 .
9% ), we use the same methods as the MLP to estimate the above statistical measures in Table 6.
[Equation]
128×
[Equation]
C 1024×
[Equation] [Equation]
[Figure] [Figure]
f
[Figure] [Figure]
[Figure] [Figure]
l ) f
[Figure] [Figure]
q
[Figure] [Figure]
E ( D q ) f 4 (13.91)
[Figure]
[Figure]
E ( E r ) f Y (1.13)
For the activation expectations of each layer in the CNN, we have x = − 0 .
023 , f 1 = − 707 . 074 , f 2 = − 6347 .
836 , f 3 = 36 . 870 , f 4 = 15 .
131 , thus all the expectations cannot be zero except the input layer, i.e., E ( X ) = 0 , E ( F 1 ) (cid:54) = 0 , E ( F 2 ) (cid:54) = 0 , E ( F 3 ) (cid:54) = 0 , E ( F 4 ) (cid:54) = 0 .
For the covariances, we show all the correlation matrices of each layer of the CNN in Figure 18.
We ﬁnd that many correlation coefﬁcients are far from zero in all the correlation matrices, thus their corresponding covariances are not close zero either.
Based on the estimations for the activation expectation and the covariance between different weights summations in each layer, we can derive that activations being independent cannot be valid for all the layers of the CNN.
For the weights summation expectation, we visualize all the expectations for each layer in the CNN in Figure 19.
We ﬁnd that some expectations have great variations, especially { E ( C l ) } Ll =1 and { E ( D q ) } Qq =1 , which means that most expectations of weights summations of different neurons in f 3 and f 4 cannot be equivalent.
As a result, the activations of the two fully connected layers in the
CNN cannot be identically distributed.
Overall, we conclude that not all the hidden layers of the CNN can satisfy the necessary conditions for activations being i.i.d.
under the assumption that the activations of the previous layer are i.i.d.
G T HE EQUIVALENCE BETWEEN THE G RADIENT DESCENT ALGORITHM AND THE FIRST ORDER APPROXIMATION
In the context of deep learning, most learning algorithms belong to the gradient descent algorithm.
Given a DNN = { x ; f 1 ; ... ; f I ; f Y } and a cost function H [ f Y , P ( Y | X )] , where P ( Y | X ) is the true distribution of the training label given the corresponding training dataset.
Let θ be the parameters of the DNN, the gradient descent aims to optimize θ by minimizing H [ f Y , P ( Y | X )]
(Rumelhart et al., 1986).
We typically update θ iteratively to derive θ ∗ , which can be expressed as
[Equation]
52) where ∇ θ t H [ f Y , P ( Y | X )] is the Jacobian matrix of H [ f Y ,
P
( Y | X )] with respect to θ t at the t th iteration, and α > 0 denotes a constant indicating the learning rate.
Since P ( Y | X ) is constant, we denote H ( f Y ) as H [ f Y , P
( Y | X )] for simplifying the following derivation.
Since the functions of all hidden layers are differentiable and the output of a hidden layer is the input of its next layer, the Jacobian matrix of H ( f Y ) with respect to the parameters of the i th hidden layer, i.e., ∇ θ ( i ) H ( f Y ) , can be expressed as follows based on the chain rule.
[Equation]
where θ ( i ) denote the parameters of the i th hidden layer.
Based on Equation 52 and Equation 53, θ ( i ) can be learned by the gradient descent method as
[Equation]
[Equation]
Table 7 summarizes the backpropagation training procedure for the MLP in Figure 15.
[Table]
If an arbitrary function f is differentiable at point p ∗ in R N and its differential is represented by the
Jacobian matrix ∇ p f , the ﬁrst order approximation of f near the point p can be formulated as
[Equation]
(55) where o ( || p − p ∗ || ) is a quantity that approaches zero much faster than || p − p ∗ || approaches zero.
Based on the equivalence between the gradient descent method and the ﬁrst order approximation (Battiti, 1992), updating the activations of the hidden layers of the MLP in Figure 15 during the backpropagation training procedure can be approximated as
[Equation] [Equation]
where f 2 [ f 1 , θ t (2)] denote the activations of the second hidden layer based on the parameters learned in the t th iteration, i.e., θ t (2) , given the activations of the ﬁrst hidden layer, i.e., f 1 .
The deﬁnitions of f 2 [ f 1 , θ t +1 (2)] and f 1 ( x , θ t (1)) are the same as f 2 [ f 1 , θ t (2)] .
Because f 2 has K neurons, i.e., f 2 = { f 2 k = σ 2 ( (cid:80) Nn =1 β nk · f 1 n + b 2 k ) }
Kk =1 , and each neuron has N + 1 parameters, i.e., θ (2) = { [ β 1 k ; · · · ; β Nk ; b
2 k ] } Kk =1 , the dimension of ∇ θ t (2) f 2 is equal to K × ( N + 1) and ∇ θ t (2) f 2 can be expressed as
[Equation]
57) where ∇ σ 2 f 2 =
∂
f 2 [ f 1 , θ t (2)] ∂σ.
Substituting ( ∇ σ 2 f 2 ) · [ f 1 ; 1] T for ∇ θ t (2) f 2 in Equation 56, we can derive f
[Equation]
If we only consider a single neuron, such as f 2 k , in f 2 , θ t +1 (2 k ) =
[ β 1 k ; · · · ; β Nk ; b 2 k ] and θ t (2 k ) =
[ β ∗ 1 k ; · · · ; β ∗ Nk ; b ∗ 2 k ] , thus [ f 1 ; 1] T · θ t +1 (2 k )
= (cid:80) Nn =1 β nk · f 1 n + b 2 k .
As a result,
Equation 58 with respect to a single neuron can be expressed as
[Equation]
Equation 59 indicates that f 2 k
[ f 1 , θ t +1 (2 k )] can be expressed as its ﬁrst order approximation with an error component based on the activation in the previous iteration, i.e., f 2 k [ f 1 , θ t (2 k )] .
Since ∇ σ 2 f 2 k
= ∂f 2 k
[ f 1 , θ t (2)]
∂σis only related to f 1 and the parameters in the t th training iteration, i.e, θ t (2) , it can be regarded as a constant.
Also note the error component not contains any parameters in the ( t + 1) th training iteration.
In summary, f 2 k ( f 1 , θ t +1 (2 k )) can be reformulated as
[Equation]
60) where C 1 = ∇ σ 2 f 2 k and C 2 = f 2 k ( f 1 , θ t (2 k )) − ( ∇
[Equation]
2 f 2 k )
[Equation]
β
[Equation]
nk
[Equation]
∗ 2 k ] .
Similarly, the activations in the ﬁrst hidden layer, i.e, f 1 , also can be formulated as the ﬁrst order approximation in the context of the gradient descent learning algorithm.
[Figure]
⋮
Output
[Figure] [Figure]
% 𝑓 ) (1) 𝑓 ) (𝐿) ⋮ 𝑓 #(
H T HE PROPOSED PROBABILISTIC REPRESENTATION HOLDS FOR CONVOLUTIONAL LAYERS
This section proves that the proposed probabilistic representation is valid for convolutional layers.
In the above CNN, the input x has Q channels x = { x q } Qq =1 (e.g., Q = 3 if x are color images), fhas N convolutional channels f 1 = { f n 1 } Nn =1 , f 2 has K convolutional channels f 2 = { f k 2 } Kk =1 .
The output layer f Y is the softmax with L nodes, thus its distribution can be formulated as
[Equation]
(61) where Z F Y =
(cid:80) Ll =1 exp ( f yl ) denotes the partition function.
Since { f yl = (cid:80)
[Equation] [Equation]
jl ·
[Equation] [Equation]
b yl } Ll =1 , we can derive
[Equation]
(62) where { f 2 j } Jj =1 is the ﬂattened output of f 2 , γ jl is the weight of the edge between f 2 j and f yl , and b yl denotes the bias.
Since f 2 is a convolutional layer with K convolutional channels, it can be reformulated as { f 2 j } Jj =1 = { f k 2 } Kk =1 .
As a result, P ( F Y ) can be reformulated as
[Equation]
where γ kl is a subset of all the parameters { γ jl } Jj =1 such that (cid:80)
Kk
=1 γ Tkl · f k
=
(cid:80)
Jj =1 γ jl · f 2 j , and γ Tkl is the transpose of γ kl .
Therefore, P ( F Y ) can be reformulated as
[Equation]
where Z (cid:48)
F Y =
Z F Y / [ exp ( b yl )] .
Recall the element-wise matrix power, e.g., exp ( a ) b , where a =
[ a 1 ; a 2 ; a 3 ] and b =
[ b 1 ; b 2 ; b 3 ] , we can derive that exp ( a ) =
[ exp ( a 1 ); exp ( a 2 ); exp ( a 3 )] and exp ( a ) b = [ exp ( a 1 ) b 1 ; exp ( a 2 ) b 2 ; exp ( a 3 ) b 3 ] =
[ exp ( a 1 b 1 ); exp ( a 2 b 2 ); exp ( a 3 b 3 )] .
As a result, exp ( ab ) = exp ( a 1 b 1 + a 2 b 2 + a 3 b 3 ) =
(cid:81) | a | exp ( a ) b , where | a | is the element numbers of a .
Based on the element-wise matrix power, we can reformulate P ( F Y ) as
[Equation]
Moreover, we introduce a new partition function Z F k=
[Equation]
exp ( f k 2 ) to guarantee that 1 Z Fk 2 exp ( f k 2 ) is a probability measure.
As a result, P ( F Y ) can be reformulated as
[Equation]
where Z (cid:48)(cid:48) F Y =
Z (cid:48)
F Y / (cid:81)
Kk =
[ Z F k 2 ] γ kl ·| f k 2 | .
Overall, P ( F Y ) can be reformulated as a PoE model, in which each expert is deﬁned by every convolutional channel in f
[Equation]
Since f 2 has K convolutional channels, i.e., f 2 = { f k 2 } Kk =1 , and each channel can be formulated as the summation of all the convolutional channels in f 1 , i.e.,
[Equation]
[Equation]
where S ( k,n )◦ f n 1 is the output of the k th convolutional ﬁlter applying into the n th channel of f 1 , b k 1 is the bias, and σ 1 ( · ) is the activation function.
If we regard all the convolutional ﬁlters for f k 2 as a clique potential function, i.e. f k 2 = ϕ k ( f 1 ) , for modeling a signal structure of the input f 1 , we can formulate P ( F 2 ) as a speciﬁc Gibbs distribution, i.e., the Markov Random Fields (MRFs), in the context of the probabilistic graphical modeling.
[Equation]
where Z F 2 = (cid:80) f 2 exp ( (cid:80) Kk =1 f k 2 ) is the partition function for P ( F 2 ) .
Based on Equation 67 and Equation 68, the distribution of an arbitrary channel in f 2 , e.g, P ( f k 2 ) , can be reformulated as
[Equation]
where Z (cid:48)
F k 2 =
Z F k 2 / exp ( b k 1 ) .
A B
C D
[Equation]
[Equation] [Equation]
$
[Equation] [Equation]
& 𝑥 ’
[Equation] [Equation]
) 𝑥 *
A B
C D
A B
C D
A B
C D
A B
C D 𝑥 " 𝑥 # 𝑥 $ 𝑥 % 𝑥 & 𝑥 ’ 𝑥 ( 𝑥 ) 𝑥 * ∘ ,
[Figure]
𝑺 𝟐(𝒌,𝒏)
Input channel 𝒇 𝟏𝒏
Output channel
[Equation]
𝟐𝒌
[Equation]
𝑦
[Equation]
𝑦 $
[Equation]
) 𝑦
[Equation]
𝑥 &
[Equation]
’ + 𝐶 , 𝑥 )
[Equation]
𝐷
[Equation]
𝑥 * = =
Blank elements are zeros 𝑦 " 𝑦 # 𝑦 $ 𝑦
[Equation]
𝑦 " 𝑦 # 𝑦 $
[Equation]
% 𝑾 𝟐(𝒌,𝒏)
Based on the equivalence between the gradient descent learning and the ﬁrst order approximation (Appendix G), we can derive
[Equation]
Based on the equivalence between a convolutional channel and a fully connected layer (Appendix
E), we can regard a convolutional operation as a matrix multiplication, which can be visualized in
Figure 21.
As a result, we have
[Equation]
where W ( k,n )is a matrix corresponding to the convolution ﬁlter S ( k,n ).
Based on the element-wise matrix power, P ( f k 2 ) can be reformulated as
[Equation]
Similarly, we introduce a new partition function Z F n=
[Equation]
n 1 exp ( f n 1 ) to guarantee that 1 Z Fn 1 exp ( f n 1 ) is a probability measure.
As a result, P ( f k 2 ) can be reformulated as
[Equation]
where Z (cid:48)(cid:48)
F k 2 =
Z (cid:48)
F k 2 / (cid:81)
Nn =1 [ Z F n 1 ] W ( k,n ).
Therefore, P ( f k 2 ) can be reformulated as a PoE model, in which all the experts are deﬁned by the convolutional channels in the previous convolutional layer.
Similar as Equation 69, the distribution of f 1 also can be expressed as a MRF model
[Equation]
where Z F 1 = (cid:80) f 1 exp ( (cid:80) Nn =1 f k 1 ) is the partition function for P ( F 1 ) .
Overall, the distribution of a convolutional layer can be formulated as a MRF model.
I T HE A PPROACH TO GENERATE THE SYNTHETIC DATASET BASED ON NIST AND THE ARCHITECTURE OF THE CNN
In this section, we present a novel approach to generate a synthetic dataset obeying the Gaussian distribution based on the NIST 1 dataset of handwritten digits by class.
The synthetic dataset has similar characteristics as the benchmark MNIST dataset.
It consists of 20,000 32 × 32 grayscale images in 10 classes (digits from 0 to 9), and each class has 1,000 training and 1,000 testing images.
More speciﬁcally, the approach to generate the synthetic dataset is summarized in Algorithm 1, and
Figure 22 visualize the spatial relation of ˆ x = { ˆ x outside , ˆ x outside-boundary , ˆ x inside-boundary , ˆ x inside } .
input: NIST dataset of handwritten digits by class 1: repeat 2: binarizing an image of NIST to obtain z 3: extracting the central part of z to obtain z c with dimension 64 × 64 4: downsampling z c to obtain z cd with dimension 32 × 32 5: extracting the edge of z cd to obtain the mask image m cd 6: decomposing m cd into four parts, i.e., m outside , m outside-boundary , m inside-boundary , and m inside .
7: sampling N (0 , 1024) to derive a random vector x ∈ R 1024 ×8: sorting x in the descending order to derive ˆ x 9: decomposing ˆ x into four parts, i.e., ˆ x = { ˆ x outside , ˆ x outside-boundary , ˆ x inside-boundary , ˆ x inside } 10: placing each pixel of ˆ x = { ˆ x outside , ˆ x outside-boundary , ˆ x inside-boundary , ˆ x inside } into the above masks.
11: until (20,000 synthetic images are generated) output: The synthetic dataset
[Figure]
The architecture of the CNN for classifying the synthetic dataset is summarized in Table 8.
[Table]
[Equation]
J A REGULARIZATION ALGORITHM BASED ON THE PROBABILISTIC REPRESENTATION J. T HE PROPOSED REGULARIZATION ALGORITHM
Based on the probabilistic representation, the ﬁrst hidden layer formulates a MRF model as a prior distribution of the training dataset and other hidden layers recursively serve as prior distributions for their respective next layer in a CNN.
However, the backpropagation calculates the gradient in the backward direction, i.e., updating the gradient of a hidden layer corresponding to a prior distribution must go over all hidden layers behind it.
In addition, the degradation problem could makes the learned prior distribution not precise enough to regularize DNNs and results in overﬁtting, especially in very deep neural networks He et al.
In this section, we propose to a new regularization learning algorithm for DNNs, which includes two steps: (i) we learn the prior distributions of DNNs directly from the training dataset { x j } Jj =1 as the blue arrow shown in Figure 23 and (ii) we use the backpropagation to train the initialized DNNs by the learned prior distributions as the red arrow shown in Figure 23.
Since directly learning prior distribution from { x j } Jj =1 is less complicated than the backpropagation learning algorithm, we can relieve the effects of degradation problem and derive prior distributions more precisely, thereby achieving the better generalization performance of DNNs.
An important question need to answer is how many hidden layers of DNNs need to be pre-trained in the ﬁrst step.
Since most prior information is stored in the training dataset { x j } Jj =1 , the hidden layer closest to the input of DNNs, i.e., the ﬁrst hidden layer f 1 , should learn the most prior information from { x j } Jj =1 as long as f 1 has powerful representation ability through including sufﬁcient hidden units, e.g., convolutional ﬁlters.
Therefore, the proposed regularization method only pre-train the ﬁrst hidden layer f 1 in order to decrease the computation complexity.
Pre-training more hidden layers could obtain better generalization performance, which can be a direction for future research.
We choose the Field of Experts (FoE) model to directly learn a prior distribution of { x j } Jj =1 because of two reasons.
Above all, FoE achieves great performance on learning prior knowledge of given datasets on various applications, e.g., image denoising Roth & Black (2005), image restoration Roth & Black (2008), and image inpainting Schmidt et al.
More importantly, FoE can be regarded as a Convolutional Neural Networks (CNN) with a single convolutional layer, hence we can easily use the learned FoE model to initialize the ﬁrst hidden layer of CNNs as long as they have the same architecture.
A FoE model can be formulated as
[Equation]
) where f k ( · ) is a linear ﬁlter, f expert k is an expert function, and Z FoE = (cid:82) x (cid:81) K k
=1 { f expert k [ f k ( x )] } d x is the partition function.
We use the contrastive divergence learning method to train p FoE ( x ) under the
Kullback-Leibler divergence (KLD) criterion (Carreira-Perpinan & Hinton, 2005).
More detail of the training procedure is in (Roth & Black, 2005; 2008).
After deriving a well-trained FoE model, we use the learned linear ﬁlters { f k } Kk =1 to initialize the ﬁrst convolutional layer of CNNs.
Finally, we use the backpropagation algorithm to train the initialized CNN for a speciﬁc application, such as image recognition.
We summarize the proposed regularization learning algorithm in Algorithm 2.
J. S IMULATIONS
In this section, we validate the proposed regularization learning algorithm for image recognition application on the CIFAR-10 benchmark dataset, which includes 70,000 32 × 32 color images in 10 classes, and each class has 6,000 training images and 1,000 testing images.
Speciﬁcally, we randomly choose 2,000 training images of each class as a new training dataset and use the same testing dataset.
Smaller training dataset means overﬁtting being more likely to occur, which can help us validate the proposed regularization learning method more convincingly.
In addition, we covert color images to grayscale to simplify computation.
In order to directly learn regularization from { x j } 20000 j =1 , we design a FoE model with 12 Gaussian
Scale Mixture (GSM) experts, thus it can be expressed as
[Equation]
[Equation]
) where f GSM k [ f k ( x )]
= (cid:80)
i =1 π ki · N ( f k ( x ); 0 , σ 2 b δ ( i ) ) , σ 2 b is a ﬁxed base variance, δ = { δ (1) , · · · , δ (11) } is a range of scales, f k ( · ) is a 3 × 3 convolutional ﬁlter, and π ki denotes the weight of each Gaussian distribution.
The parameters of the FoE model is θ = { f, π } , where f = [ f 1 , ..., f 12 ] and π = { π ki } k =12 ,i =11 k
=1 ,i =1 .
[Table]
The architecture of the CNN for image recognition is summarized in Table 9.
It is noteworthy that the number of convolutional ﬁlters in f 1 and the ﬁlter dimension are the same as the above
FoE model, thus we can use the well-trained FoE to initialize f 1 as a regularization for the CNN classifying the CIFAR-10 dataset based on Algorithm 2.
In order to make a comprehensive comparison, we choose two commonly used regularizations, i.e., the dropout (drop rate is 0.3) and the L 2 norm, as references.
We use the testing error to exam the generalization performances of different regularizations.
Figure 25 visualizes that the proposed regularization method (abbr.
Bayes Prior (BP)) successfully decreases the testing error.
In particular, it outperforms dropout ( 32 . 26% ) and achieve similar performance as L 2 norm ( 29 .
Therefore, this experiment validates the effectiveness of the proposed regularization learning method.
[Figure]
Testing error of CNN without any regularization (33.95%)
Testing error of CNN with droupout (32.26%)
Testing error of CNN
[Figure]
[Figure]
Moreover, the proposed regularization method can be easily combined with other regularizations methods to improve the generalization performance further.
We combine two arbitrary aforementioned regularization methods together and examine their generalization performances under the same experimental conditions as above.
Figure 26 shows that the combination of two regularizations together achieves better generalization performance than any single one.
In particular, the combination of BP and L2 norm achieves the lowest testing error ( 28 . 60% ).
Also note that if we combine three regularizations together, we achieve the best generalization performance ( 27 .
This experiment further validate the proposed regularization learning algorithm.
[Figure]
Testing error of CNN with Bayes Prior (BP) (29.90%)
Testing error of CNN with droupout + L2 norm (29.29%)
Testing error of CNN with BP + droupout (28.91%)
Testing error of CNN with BP + L2 norm (28.60%)
Testing error of CNN with BP + both (27.69%)