P ROBABILISTIC MODELING THE HIDDEN LAYERS OF D EEP N EURAL N ETWORKS
Anonymous authors
Paper under double-blind review
A BSTRACT
In this paper, we demonstrate that the parameters of Deep Neural Networks (DNNs) do not satisfy the i.i.d.
prior assumption and activations being i.i.d. is not valid for all the hidden layers of DNNs.
Therefore, the Gaussian process cannot correctly model the distribution of the hidden layers.
Alternatively, we propose a novel probabilistic representation for the hidden layers of DNNs in two aspects: (i) a hidden layer formulates a Gibbs distribution, of which the energy function depends on the activations, and (ii) the statistical connection between two adjacent layers can be formulated as a product of experts model.
Moreover, we conﬁrm the equivalence between the stochastic gradient descent and the variational inference and prove the former is also equivalent to the energy minimization optimization.
Based on the proposed probabilistic representation, we show that the entire architecture of DNNs can be explained as a Bayesian hierarchical model and DNNs have explicitly Bayesian regularizations deﬁned by the hidden layers serving as prior distributions.
Simulation results validate the proposed theories.
I NTRODUCTION
Recently, interpreting the hidden layers of Deep Neural Networks (DNNs) as the Gaussian Process (GP) has attracted a great deal of attention because it provides a novel probabilistic perspective to clarify the working mechanism of deep learning.
Neal (1994) initially demonstrates the equivalence between GP and neural networks with a single fully connected layer in the limit of inﬁnite neurons.
Following this seminal work, Lee et al.
; Matthews et al.
(2018) further extend the equivalence to neural networks with multiple hidden layers and Garriga-Alonso et al.
; Novak et al.
(2018) establish the correspondence between the convolutional layers and GP.
It is important to note that the GP explanations rely on a fundamental probabilistic premise that the activations of a hidden layer are independent and identically distributed (i.i.d.).
More speciﬁcally, based on the classical Central Limit Theorem (CLT), the prerequisite of an activation of a hidden layer approaching a Gaussian distribution is that all the activations of the previous layer are i.i.d..
Though the Lyapunov CLT could relax the probabilistic premise to be independence only, all the previous works do not thoroughly discuss the probabilistic premise except assuming an i.i.d.
prior for all the parameters of DNNs.
Due to its extreme importance but unclear status, it is necessary to thoroughly investigate the probabilistic premise for improving the interpretability of DNNs.
In this work, we demonstrate that the parameters of DNNs are correlated and depend on the training dataset, i.e., they do not satisfy the i.i.d.
prior.
Moreover, we demonstrate that activations being i.i.d.
cannot hold for all the hidden layers of DNNs.
In the context of Bayesian probability, we theoretically derive the necessary conditions for the activations of a hidden layer being i.i.d.
given the assumption that the activations of the previous layer are i.i.d..
Subsequently, we experimentally show that typical DNNs, such as the Multilayer Perceptron (MLP) and the Convolutional Neural
Networks (CNNs), cannot satisfy the necessary conditions.
As a result, activations being i.i.d. is not valid for all hidden layers.
In other words, GP with the i.i.d.
assumption for the parameters of DNNs cannot correctly explain all the hidden layers of DNNs.
In addition, some previous works show that
GP is sensitive to the curse of dimensionality (Bengio et al., 2005; Hinton et al., 2012) and cannot clarify the hierarchical representation, an essential of deep learning (Matthews et al., 2018).
Alternatively, we propose a novel probabilistic representation for the hidden layers based on the
Gibbs distribution (LeCun et al., 2006) and the Product of Experts (PoE) model (Hinton, 2002).
The proposed probabilistic representation explains a hidden layer in two aspects: (i) the distribution of a hidden layer can be formulated as a Gibbs distribution, of which the energy function is determined by the activations, and (ii) the connection between two adjacent layers can be modeled by a PoE model.
Speciﬁcally, the distribution of a single neuron or hidden unit (e.g., convolutional channel) can be expressed as a PoE, in which all the experts are deﬁned by the neurons or hidden units in the previous hidden layer.
Compared to the GP explanations, the proposed probabilistic representation provides a more speciﬁc explanation for the hidden layers of DNNs without any assumption.
Moreover, we conﬁrm the equivalence between the variational inference and the stochastic gradient descent (SGD) (Mandt et al., 2017; Chaudhari & Soatto, 2018), and demonstrate the later is also equivalent to the energy minimization optimization, which in turn validates the Gibbs explanation.
Based on the probabilistic representation, the entire architecture of DNNs can be explained as a
Bayesian Hierarchical Model (BHM) (Li & Perona, 2005).
Precisely, the output layer formulates a likelihood distribution, each hidden layers recursively serve as prior distribution for the next layer, and the ﬁrst hidden layer formulates a prior distribution of the training dataset.
That provides a clearly probabilistic explanation for the hierarchical property of DNNs. Moreover, in contrast to recent works claiming that SGD performs implicit regularization (Zhang et al., 2016; Neyshabur et al., 2017), the BHM explanation indicates explicitly Bayesian regularizations in DNNs because regularizations correspond to prior distributions in terms of Bayesian regularization theory (Steck & Jaakkola, 2003).
Finally, we propose a novel regularization approach to improve the generalization performance of DNNs by pre-training the hidden layers corresponding to prior distributions.
P RELIMINARIES
We assume X
and Y are two random variables and P θ ( X , Y )
=
P ( Y | X ) P ( X ) is an unknown joint distribution between X and Y , where P ( X ) describes the prior knowledge of X , P ( Y | X ) describes the statistical connection between X and Y , and θ indicate the parameters of P θ ( X , Y ) .
A dataset D = { ( x j , y j ) | x j ∈ R M , y j ∈ R L } Jj =1 is composed of i.i.d.
samples generated from
P θ ( X , Y ) .
A neural network with I hidden layers is denoted as DNN = { x ; f 1 ; ... ; f I ; f Y } and trained by D , where ( x , y ) ∈
D are the input of the DNN and the corresponding training label, and f Y is an estimation of the true distribution P ( Y | X ) .
As a result, x ∼ P ( X ) and f Y ≈ P
( Y | X ) .
The meaning of f i is two-fold: (i) denoting all the neurons in the i th hidden layer, and (ii) representing the activations of the i th hidden layer.
In addition, F i and F Y are the random variables corresponding to f i and f Y , respectively.
In this work, we use the MLP (Figure 1) to derive most theoretical conclusions unless otherwise speciﬁed.
𝑓 "" 𝑥 " 𝒙 𝑥 % 𝑥 & 𝑓 "’ 𝑓 "( 𝑓 ) (1) 𝛽 ’" 𝛽 "" 𝛽 (" 𝛼 "’ 𝛼 %’ 𝛼 &’ 𝑓 "/ 𝛽 /" ⋮ ⋮ ⋮ ⋮ 𝒇𝒇 𝒀 𝑓 ) (𝐿) 𝑓 ’" 𝑓 ’’ 𝑓 ’ ⋮ 𝒇𝛾 "8 𝛾 ’ 𝛾 the n th linear ﬁlter, α mn is the weight of the edge between x m and f 1 n , b 1 n is the bias, and σ 1 is a non-linear activation function.
Similarly, f 2 = { σ 2 [ g 2 k ( f 1 )] } Kk =1 where g 2 k ( f 1 ) =
(cid:80) Nn =1 β nk · f 1 n + b 2 k .
In addition, since f Y is the softmax, f Y
= { 1 Z Y exp [ g yl ( f 2 )] } Ll =1 where g yl ( f 2 ) =
(cid:80)
Kk =1 γ kl · f 2 k + b yl , and the partition function is Z Y =
(cid:80) Ll =1 exp [ g yl ( f 2 )] .
B ACKGROUND
As a fundamental probabilistic model, Gibbs distribution (a.k.a., energy based model, Boltzmann distribution, or renormalization group) formulates the dependence of random variables X through associating an energy to each dependence structure (Geman & Geman, 1984).
[Equation]
where E ( x ; θ ) is the energy function, θ denotes the parameters, and Z ( θ )
= (cid:82) x exp [ − E ( x ; θ )]
d x is the partition function.
Especially, a Gibbs distribution can be easily reformulated as various probabilistic models by redeﬁning E ( x ; θ ) , e.g., it would become Markov Random Fields (MRFs) model if the energy function is deﬁned as the summation of multiple potential functions, i.e.,
E ( x ; θ ) = − (cid:80) k f k ( x ; θ k ) (Wang et al., 2013; Goodfellow et al., 2016).
Moreover, assuming a single potential function deﬁnes an expert, i.e., F k = exp [ − f k ( x ; θ k )] , we can regard MRFs as a PoE model, i.e., P ( x ; θ ) =
1 Z ( θ ) (cid:81)
k F k , where Z ( θ ) = (cid:81)
k Z ( θ k ) (Hinton, 2002).
A Gibbs distribution has two appealing properties.
First, E ( x ; θ ) is the sufﬁcient statistics of
P ( x ; θ ) because Z ( θ ) only relies on θ , i.e., P ( x ; θ ) entirely depends on E ( x ; θ ) .
The property establishes a connection between P ( X ; θ ) and the deterministic function E ( x ; θ ) , which allow us to explain a deterministic hidden layers in a probabilistic way.
Second, the energy minimization is the commonly used optimization for learning θ , i.e., θ ∗ = arg min θ E ( x ; θ ) (LeCun et al., 2006), which is helpful to explain the training procedure of DNNs, because the energy minimization can be implemented by the gradient descent algorithm as long as E ( x ; θ ) is differentiable.
Numerous efforts have been devoted to explain DNNs from the viewpoint of Gibbs distribution.
The classical work is the Restricted Boltzmann Machine (RBM) (Salakhutdinov & Hinton, 2009), which is a simpliﬁed Gibbs distribution.
Speciﬁcally, RBM only considers the connections between visible units x and hidden units h but overlooks the internal connections within x and h , thus it cannot explain complicated DNNs, e.g., CNNs.
To the best of our knowledge, Mehta & Schwab (2014) ﬁrst explain the distribution of hidden layers as a Gibbs distribution, and Yaida (2019) indirectly proves the distribution of fully connected layers is a Gibbs distribution.
Lin et al.
(2017) clarify some properties of DNNs based on Gibbs distribution.
However, all previous works still cannot derive an explicitly Gibbs explanation for complex hidden layers, e.g., convolutional layers.
M AIN RESULTS 4. T HE PARAMETERS OF DNN S DO NOT SATISFY THE I . I . D . ASSUMPTION
We examine the statistical properties of the parameters of DNNs and show they are correlated.
First, we derive a necessary condition for the parameters of DNNs satisfying the i.i.d.
prior, Second, we experimentally show that typical DNNs cannot satisfy the necessary condition on the MNIST and
CIFAR-10 datasets.
Hence, the parameters of DNNs do not satisfy the i.i.d. assumption.
If all the parameters of DNNs are i.i.d., the parameters of each neuron are identically distributed, e.g., { α mn } Mm =1 ∼ P ( α n ) in the MLP (Figure 1).
Since all the parameters are i.i.d., different neurons should be uncorrelated, i.e., ∀ n (cid:54) = n (cid:48) , Corr ( α n , α n (cid:48) )
= 0 .
In particular, the i.i.d.
assumption enables us to use the sample correlation r ( α n , α n (cid:48) ) to estimate Corr ( α n , α n (cid:48) ) .
[Equation]
where α n = 1 M (cid:80) Mm =1 α mn is the sample mean of { α mn } Mm =1 .
Therefore, taking into account the estimation error, | r ( α n , α n (cid:48) ) | close to zero is the necessary condition for all parameters being i.i.d..
We examine the necessary condition for all parameters being i.i.d.
in various DNNs.
First, we use the MLP to classify the benchmark MNIST dataset.
The information about the network architecture and training methods is included in Appendix A.1.
After training the MLP well, we calculate the absolute sample correlations, namely, | r ( α n , α n (cid:48) ) | , | r ( β k , β k (cid:48) ) | , and | r ( γ l , γ l (cid:48) )
| , and derive three absolute correlation matrices, i.e., A (cid:48)
N × N , B (cid:48)
K × K , and C (cid:48)
L × L , shown in Figure 2.
We ﬁnd that most | r ( α n , α n (cid:48) ) | are close to zero, but most | r ( β k , β k (cid:48) )
| and | r ( γ l , γ l (cid:48) )
| are non-zero.
That indicates the necessary condition cannot hold for all the parameters of the MLP.
[Figure]
Second, we construct a shallow neural network with a single fully connected hidden layer, namely,
NN = { x ; f ; f Y } , for classifying the MNIST dataset based on the seminal work (Neal, 1994) in
Appendix A.2, and construct a CNN for classifying the CIFAR-10 dataset in Appendix A.3.
Using the same method in the MLP, we obtain the absolute sample correlations between the weights of different nodes in the both neural networks.
The results in Appendix A.2 and Appendix A.3 show that the weights of DNNs do not satisfy the necessary condition.
Third, we visualize all the parameters of the NN = { x ; f ; f Y } for classifying a synthetic dataset in Appendix A.4.
Since the synthetic dataset is much simpler than MNIST, we can directly show the correlation of different parameters and their dependences on the dataset by visualizing them.
Overall, these simulations demonstrate that the parameters of DNNs are not i.i.d..
4. A CTIVATIONS BEING I . I . D . IS NOT VALID FOR ALL THE HIDDEN LAYERS OF DNN S
Although we demonstrate that the parameters of DNNs do not satisfy the i.i.d.
assumption, we are still unclear whether the activations of a hidden layer are i.i.d..
In this section, we demonstrate that activations being i.i.d.
is not valid for all the hidden layers of DNNs.
In the context of Bayesian probability, all the weights and biases of the MLP (Figure 1) have prior distributions, i.e., α mn ∼ P ( A mn ) , β nk ∼ P
( W nk ) , b 1 n ∼ P
( B 1 n ) , and b 2 k ∼ P ( B 2 k ) , thus we regard G 1 n = (cid:80)
Mm =
A mn X m + B 1 n as the random variable of g 1 n ( x ) = (cid:80)
Mm =1 α mn · x m + b 1 n , and G 2 k
= (cid:80) Nn
=1 W nk F 1 n + B 2 k as the random variable of g 2 k ( f 1 ) =
(cid:80) Nn =1 β nk · f 1 n + b 2 k .
Hence, the random variables of two arbitrary activations in f 1 and f 2 , e.g., f 1 n and f 2 k , can be expressed as F 1 n = σ 1 ( G 1 n ) and F 2 k = σ 2 ( G 2 k ) , respectively.
Given two random variables A and B , the necessary conditions for them being i.i.d.
are that they are uncorrelated (i.e., Cov ( A, B ) = 0 ) and have the same expectation (i.e., E ( A )
=
E ( B ) ).
Similarly, assuming { F 1 n } Nn =1 are i.i.d.
and { f 1 n } Nn =1 ∼ P ( F 1 ) , we derive the following necessary conditions for the activations of the second hidden layer, i.e. , { F 2 k } Kk =1 , being i.i.d..
∀ ( k, k (cid:48) )
∈ S 1 = { ( k, k (cid:48) )
∈
Z 2 | k (cid:54)
=
k (cid:48) , 1 ≤ k ≤ K, 1 ≤ k (cid:48) ≤ K } , we must have
[Equation]
σ 2 ( · ) is strictly increasing, differentiable, and invertible.
[Equation]
where W k = (cid:80) Nn =1 W nk .
Equation 3 and 4 are the necessary conditions for { G 2 k } Kk =
being i.i.d..
Equation 5 constrains the activation function σ 2 ( · ) to guarantee that if { G 2 k } Kk =1 are i.i.d.
then { F 2 k } Kk =1 are i.i.d.
as well.
The necessary conditions are valid for arbitrary fully connected layers as long as properly changing the subscripts.
The detailed derivations are included in Appendix B and
Appendix C. Moreover, we prove that the necessary conditions also hold for convolutional layers based on the connection between fully connected layers and convolutional layers in Appendix E.
[Table]
It is noteworthy that the i.i.d.
assumption for the parameters of DNNs indeed satisfy the necessary conditions for activations being i.i.d..
Nevertheless, since the i.i.d.
prior is not an appropriate prior for the parameters of DNNs, it cannot guarantee activations of a hidden layer being i.i.d..
The theoretical necessary conditions indicate an applicable approach to examine whether activations of a hidden layer being i.i.d.
given the assumption that the activations of the previous layer are i.i.d..
More speciﬁcally, since { F 1 n } Nn =1 are i.i.d.
, we can use the sample mean f 1 to estimate E ( F 1 ) .
Because the samples of the training dataset D are also assumed to be i.i.d.
, we can train the MLP multiple times to draw multiple independent observations of W nk and B 2 k , thus we can use the sample covariance to estimate Cov ( W k , W k (cid:48) ) ,
Cov ( W k , B 2 k (cid:48) ) , and etc.
Based on the empirical approach, we estimate the statistical measures (Table 1) to examine whether the activations of the three layers (i.e., f 1 , f 2 , and f Y ) being i.i.d.
in the MLP for classifying the
MNIST dataset.
Without loss of generality, we restrict all the layers of the MLP from using bias to decrease computation complexity, thus the necessary conditions for f 2 can be simpliﬁed as
[Equation]
we must
[Equation]
First, Figure 3 shows the correlation matrices, i.e., A N × N , W K × K , and C L × L , to check if their corresponding covariances, i.e, Cov ( A n , A n (cid:48) ) , Cov ( W k , W k (cid:48) ) , and Cov ( C k , C k (cid:48) ) , are close to zero.
It shows that many correlations are far from zero, so their corresponding covariances are not close to zero either.
Second, we use the sample mean, e.g., x , to estimate the expectation, e.g., E ( X ) .
Taking into account the estimation error, we can regard E ( X ) =
based on x = 0 .
However, f 1 = 0 .
436 and f 2 = 0 .
498 imply E ( F 1 ) (cid:54) = 0 and E ( F 2 ) (cid:54) = 0 even considering the estimation error.
Therefore, the activations of the last two hidden layer, i.e., f 2 and f Y , cannot be independent because they cannot satisfy the ﬁrst necessary condition (Equation 6).
In addition, we show the necessary condition for activations being identically distributed also cannot be satisﬁed by the MLP in Appendix F.1.
Furthermore, we demonstrate that the necessary conditions for activations being i.i.d.
also cannot be satisﬁed in the CNN for classifying the CIFAR-10 dataset in Appendix F.
[Figure]
Overall, we conclude that activations being i.i.d.
is not valid for all the hidden layers of DNNs based on the theoretical necessary conditions and the experimental simulations on typical DNNs.
In other words, we cannot use the CLT to establish the equivalence between GP and all the hidden layers of
DNNs, thus GP can not correctly clarify all the hidden layers of DNNs.
That necessitates a more general probabilistic representation for explaining the hidden layers of DNNs.
4. T HE DISTRIBUTION OF A HIDDEN LAYER IS A G IBBS DISTRIBUTION
This section proves that the distribution of a hidden layer can be formulated as a Gibbs distribution and the statistical connection between two adjacent layers can be modeled as a PoE model.
Assuming the output layer in the MLP (Figure 1) is the softmax, its distribution can be expressed as
[Equation]
where Z Y =
(cid:80) Ll =1 exp ( f yl ) is the partition function.
Since f yl = (cid:80) Kk =1 γ kl · f 2 k + b yl , we have
[Equation]
Based on the properties of the exponential function, i.e., exp ( a + b ) = exp ( a ) · exp ( b ) and exp ( a · b ) =
[ exp ( b )] a , we can reformulate P ( F Y ) as
[Equation]
where Z (cid:48)
Y =
Z Y / exp ( b yl ) .
Since { exp ( f 2 k ) } Kk =1 are scalar, we can introduce a new partition function Z F 2 = (cid:80)
Kk =1 exp ( f 2 k ) such that { 1 Z F 2 exp ( f 2 k ) }
Kk =1 become a probability measure.
As a result, we can further reformulate P ( F Y ) =
{ P ( f yl ) } Ll =1 as a Product of Expert (PoE) model as
[Equation]
where Z (cid:48)(cid:48) Y =
Z Y / [ exp ( b yl ) · (cid:81)
Kk =
[ Z F 2 ] γ kl ] and each expert is deﬁned as 1 Z F 2 exp ( f 2 k ) .
It is noteworthy that all the experts { 1 Z F 2 exp ( f 2 k ) } Kk =1 form a probability measure and establish an exact one-to-one correspondence to all the neurons in the second hidden layer f 2 , i.e., { f 2 k } Kk =1 .
Therefore, the distribution of f 2 can be expressed as
[Equation]
Based on the deﬁnition of Gibbs distribution, Equation 8 and 12 show that f Y and f 2 formulate two multivariate Gibbs distributions and their energy functions are equivalent to the negative of the output nodes { f yl } Ll =1 and the neurons { f 2 k } Kk =1 , respectively.
In addition, Equation 11 implies that the connection between P ( f yl ) and P ( f 2 k ) can be expressed as the PoE model, namely that P ( f yl ) is equal to the product of all the experts { P ( f 2 k ) }
Kk =1 with their respective weights { γ kl } Kk =1 .
Moreover, we prove that the probabilistic representation is valid for other hidden layers in the MLP.
Since { f 2 k = σ 2 ( (cid:80) Nn =1 β nk · f 1 n + b 2 k ) }
Kk =1 , P ( F 2 ) can be expressed as
[Equation]
Based on the equivalence between the gradient descent algorithm and the ﬁrst order approximation (Battiti, 1992), we prove that σ 2 ( (cid:80) Nn =1 β nk · f 1 n + b 2 k )] can be approximated as
[Equation]
where C 21 and C 22 only depend on the activations { f 1 n } Nn =1 in the previous training iteration, thus they can be regarded as constants and absorbed by β nk and b 2 k .
The proof for the approximation is included in Appendix G. Therefore, P ( F 2 ) also can be modeled as a PoE model.
[Equation]
where Z
[Equation]
[ Z
[Equation] [Equation]
] and Z F
[Equation]
= (cid:80) Nn
[Equation]
exp ( f
[Equation]
Finally, the distribution of the ﬁrst hidden layer f 1 is formulated as
[Equation]
Overall, we introduce a novel probabilistic representation for fully connected layers in two aspects.
First, a fully connected layer formulates a multivariate Gibbs distribution (Equation 8, 12, and 16) and the energy functions is equivalent to the negative activation.
Second, the connection between two adjacent fully connected layers can be formulated as a PoE model (Equation 11 and 15).
The proposed probabilistic representation has two advantages over the GP explanations.
First, the
FoE explanation does not require any assumption, e.g., we formulate P ( f 2 k ) as a PoE model, in which all the experts { 1 Z F 1 exp ( f 1 n ) }
Nn =1 are deﬁned by all the neurons { f 1 n } Nn =1 .
As a comparison, the GP explanation derives P ( f 2 k ) as a Gaussian distribution in the limit of inﬁnite neurons, i.e., { f 1 n } N →∞ n =1 , under the assumption that { f 1 n } Nn =1 are i.i.d..
Second, the Gibbs distribution provides a more explicit explanation for the functionality of a fully connected layer.
Speciﬁcally, a Gibbs distribution derives the probability of the multiple features deﬁned by the neurons occurring in the input, but the covariance function of a GP cannot clearly explain the functionality of each neuron.
Moreover, we prove that a convolutional layer formulates a speciﬁc Gibbs distribution, namely the MRF model, in Appendix H. More speciﬁcally, a convolutional layer
f
i with K convolutional channels, i.e., f i = { f ki =
σ i ( (cid:80) Nn =1 S ( k,n ) i
[Equation]
f ni
[Equation] [Equation]
b ki
[Equation]
) } Kk =1 , formulate a MRF model as
[Equation]
where f i − 1 = { f ni − 1 } Nn =1 is the input, ϕ ki
( f i − 1 )
= (cid:80) Nn =1 S ( k,n ) ◦ f ni − 1 + b ki · 1 ) is the k th linear convolutional channel, S ( k,n ) is the k th convolutional ﬁlter applying to the n th input channel, b ki is the bias, σ i ( · ) is an non-linear operator, and the partition function Z F i =
[Equation]
exp [
[Equation]
K k =1 ϕ k
[Equation]
f i − 1 )]
d f i − 1 .
The corresponding energy function E Fi = − (cid:80)
Kk =1 ϕ k
( f i − 1 ) .
In summary, the distribution of a hidden layer f i can be formulated as a Gibbs distribution P ( F i ) , especially the energy function E F i establish a connection between the deterministic function of the hidden layer and the Gibbs distribution P ( F i ) .
In addition, the correspondence between Gibbs distributions and hidden layers is summarized in Table 2.
[Table]
[Equation]
Explanation energy function discrete
[Table]
energy function
[Equation]
We use Equation 16 and Equation 17 as examples to illustrate the correspondence.
S IMULATIONS
This section uses two synthetic datasets to demonstrate the proposed probabilistic representation: (i) the distribution of a fully connected layer is a multivariate discrete Gibbs distribution and (ii) the distribution a convolutional layer is a MRF model.
5. T HE DISTRIBUTION OF A FULLY CONNECTED LAYER IS A MULTIVARIATE DISCRETE G IBBS DISTRIBUTION
A fully connected layer with N neurons, i.e., f = { f n } Nn =1 , formulates a multivariate discrete
Gibbs distribution to describe the probability of the N features deﬁned by their respective neurons occurring in the input.
The energy function is equal to the negative activation, i.e., E f n = − f n ( x ) .
[Equation]
(18) where f n ( x ) =
σ [ g n ( x )] , g n ( x )= (cid:80)
Mm =1 α mn · x m + b n is a linear ﬁlter with α mn being the weights and b n being the bias, σ ( · ) is the activation function, and Z F = (cid:80) N n =1 exp
[ f n ( x )] is the partition function.
Since benchmark datasets consist of very complex features and typical DNNs have too complicated architecture, it is very hard to directly demonstrate the Gibbs distribution explanation.
Alternatively, we choose the neural network for classifying the synthetic dataset in Appendix A.4, because the neural network only has a single fully connected hidden layer with seven neurons, i.e., N = 7 , and the synthetic dataset only has four features.
After the network is trained well, we choose a synthetic image as the input and show all the information of the learned hidden layer in Figure 4.
Since the input image is sorted in the primary diagonal direction, the input feature can be described as high values above the secondary diagonal (the red line) and low values below the red line.
Figure 4 shows that Neuron1, Neuron 2 and Neuron 4 correctly describe the input feature, thus they derive large positive outputs (i.e., g 1 ( x ) = 122 . 09 , g 2 ( x ) = 113 . 70 , and g 4 ( x ) = 140 . 20 ).
However,
Neuron 6 and Neuron 7 do not describe the input feature correctly, thus they derive large negative outputs (i.e., g 6 ( x )
= − 134 .
08 , g 7 ( x ) = − 133 . 36 ).
After applying the sigmoid function, we have f 1 ( x ) =
f 2 ( x ) =
f 4 ( x ) = 1 and f 6 ( x )
=
f 7 ( x ) = 0 .
Finally, we derive the probability of different neurons, i.e., P ( f 1 ) =
P ( f 2 ) = P ( f 4 ) = 0 .
20 and P ( f 6 ) =
P ( f 7 ) = 0 . 07 , which indicates that the features deﬁned by Neuron 1, Neuron 2 and Neuron 4 have high probability occurring in the input but the features deﬁned by Neuron 6 and Neuron 7 have low probability occurring in the input.
[Figure]
[Figure] [Figure]
[Figure] [Figure]
5. T HE DISTRIBUTION OF A CONVOLUTIONAL LAYER IS A MRF MODEL
Since the distributions of benchmark datasets are unknown, it is impossible to use them to verify the probabilistic explanation for convolutional layers.
Alternatively, we generate a synthetic dataset obeying the Gaussian distribution N (0 , 1024) based on the NIST dataset of handwritten digits.
It consists of 20,000 32 × 32 grayscale images in 10 classes (digits from 0 to 9), and each class has 1,000 training images and 1,000 testing images.
Figure 5 shows ﬁve synthetic images and their perspective histograms.
The method for generating the synthetic dataset is included in Appendix I.
We design a CNN = { x ; f 1 ; f 2 ; f 3 ; f 4 ; f Y } for classifying the synthetic dataset.
The CNN has two convolutional layers with ReLU (i.e., f 1 and f 3 ), two max-pooling layers (i.e., f 2 and f 4 ) and a softmax output layer f Y .
The detailed information about the CNN is included in Appendix I.
Based on the MRF explanation (Equation 17), the distribution of f 1 can be formulated as
[Equation]
where ϕ k 1 ( x ) =
S k ◦ x + b k 1 · 1 is the k th linear convolutional channel, S k is the linear convolutional ﬁlter and b k 1 is the bias.
The energy function of P ( F 1 ) is equivalent to E F 1 = − (cid:80) 20 k =1 f k 1 .
In order to model a high dimensional dataset, e.g., the 32 × 32 synthetic image, we typically need a multivariate distribution with the same dimension, e.g., the dimension of the covariance function of a Gaussian process should be 1024 × 1024 .
However, most structured probabilistic graphical models, especially the MRF model, have two fundamental assumptions: stationary and Markov (Lyu & Simoncelli, 2007).
The former means that the distribution is independent on the location, i.e., the distribution of pixels at different locations are identically distributed.
The later indicates that the distribution of a single pixel is independent on the other pixels given its neighbors.
The two assumptions allow us to use the histogram of samples to simulate the true distribution.
[Figure] [Figure]
After the CNN is well trained, we randomly choose a testing image x as the input to derive P ( F 1 ) .
Since x ∼ P ( X ) , we can use the histogram of the testing image, i.e., P ( x ) , to estimate P ( X ) .
In addition, E F 1 is a sufﬁcient statistics for P ( F 1 ) such that we can use the histogram of E Fto estimate P ( F 1 ) .
As a result, we can verify the probabilistic representation by calculating the distance between P ( F 1 ) and P ( X ) , i..
e, KL [ P ( X ) || P ( F 1 )] .
We visualize their distributions in
Figure 6, which shows that P ( F 1 ) is very close to P ( X ) , i.e., KL ( P ( X ) ||
q ( F 1 )) = 0 . 83 , which demonstrates the distribution of a convolutional layer can be modeled as a MRF model.
In addition, the experiment provides a new viewpoint to demonstrate the activations of different convolutional channels are not i.i.d..
Given another synthetic image, we visualize the output of ﬁve convolutional channels in f 1 in Figure 7.
It shows that the activations of different channels at the same position are correlated, i.e., they are not independent.
We also estimate the distribution of each convolutional channel based on the histograms of their respective energy functions.
It shows that the distributions of different channels are different, i.e., they are not identically distributed.
D ISCUSSION
Based on the proposed probabilistic explanations for the hidden layer of DNNs, we discuss certain theoretical and practical topics of deep learning.
First, we demonstrate that the entire architecture of DNNs can be explained as BHM, which derives a probabilistic explanation for the hierarchical property of DNNs.
Second, we further conﬁrm the equivalence between SGD and the variational inference.
Third, we demonstrate explicitly Bayesian regularizations in DNNs and propose a novel regularization method to improve the generalization performance of DNNs.
6. T HE ENTIRE ARCHITECTURE OF DNN S CAN BE EXPLAINED AS A BHM
Since the output of the hidden layer f i in a DNN = { x ; f 1 ; ... ; f I ; f Y } is commonly the input of the next layer
f i +1 , the DNN formulates a Markov chain F 1 → · · · → F I → F Y .
As a result, the distribution of a hidden layer in the DNN should be regarded as a conditional distribution and the entire architecture of the DNN can be formulated as a joint distribution.
[Equation]
Since P ( F Y | F I ) is an estimation of the true distribution P ( Y | X ) , it can be regarded as a likelihood distribution.
Simulations in Section 5 show that P ( F 1 | X ) works as a prior distribution of the training dataset.
The Markov chain indicates that other hidden layers recursively serve as prior distributions for their respective next layers, i.e., P ( F i +
| F i ) .
As a result, the entire architecture of
DNNs can be explained as a BHM.
That provides a novel probabilistic explanation for the hierarchy property of DNNs, which cannot be clariﬁed by GP (Matthews et al., 2018).
6. SGD CAN BE EXPLAINED AS A VARIATIONAL INFERENCE
Recent works demonstrate that SGD works as a speciﬁc Bayesian posterior inference, namely the variational inference (Mandt et al., 2017; Chaudhari & Soatto, 2018).
Here we further conﬁrm the
Bayesian explanation for SGD in Appendix J based on the proposed probabilistic representation.
Brieﬂy speaking, given a DNN = { x ; f 1 ; ... ; f I ; f Y } , we prove that the conditional distribution of f Y given x , i.e., P ( F Y | X ) , is also a Gibbs distribution and its energy function is equal to the deterministic function corresponding to the entire architecture of DNNs, i.e., DNNs describes a family of Gibbs distribution Q .
Based on the deﬁnition of variational inference, SGD aims to ﬁnd a Gibbs distribution P ∗ ( F Y | X ) such that it can minimize the distance between P ( F Y | X ) and the true posterior distribution P ( Y | X ) determined by the training dataset D .
[Equation]
Moreover, we prove that SGD is also equivalent to the energy minimization optimization, which in turn validates the Gibbs explanation for the hidden layer of DNNs.
We demonstrate the theoretical explanation for SGD based on the CNN classifying the synthetic dataset and the MLP classifying the MNIST dataset in Appendix K.1.
In addition, we demonstrate that the learned parameters via
SGD are not i.i.d.
based on in Appendix K.2, which indirectly justiﬁes that we cannot correctly clarify DNNs based on the existing GP explanations.
6. E XPLICITLY B AYESIAN REGULARIZATION IN DNN S
Most recent works ascribe the generalization performance of DNNs to SGD performing an implicit regularization, i.e., a bias towards models with low complexity (Keskar et al., 2016; Gunasekar et al., 2018; Soudry et al., 2018).
However, the implicit regularization have lots of controversies (Dinh et al., 2017; Arora et al., 2019).
Alternatively, the BHM explanation indicates explicitly Bayesian regularization in DNNs because a regularization corresponds to a prior distribution in the context of Bayesian theory (Steck & Jaakkola, 2003).
In particular, the Bayesian regularization answers some problems of DNNs that cannot be clariﬁed by the implicit regularization, e.g., increasing the number of hidden units can decrease the generalization error rather than leading to overﬁtting.
Simulations in Appendix K.3 demonstrate that SGD indeed learn an explicitly Bayesian regularization and clarify the above problem of deep learning that cannot be explained by the implicit regularization.
Moreover, the BHM explanation for DNNs indicates a novel regularization approach to improve the generalization performance of DNNs through pre-training the hidden layers corresponding to prior distributions.
In particular, we introduce a novel regularization algorithm for CNNs and demonstrate its superiority over traditional regularizations, e.g., dropout (Srivastava et al., 2014), based on the image recognition task on the benchmark CIFAR-10 dataset in Appendix L.
C ONCLUSION
In this paper, we demonstrate that the parameters of DNN do not satisfy the i.i.d assumption and the activations being i.i.d.
cannot be valid for all the hidden layers of DNNs.
Hence, GP cannot correctly explain all the hidden layer of DNNs.
Alternatively, we propose a novel probabilistic representation for DNNs in tree aspects: (i) a hidden layer formulates a Gibbs distribution, of which the energy function is determined by the activations, (ii) the connection between two adjacent layers can be modeled by a PoE model, and (iii) the entire architecture of DNNs can be explained as a BHM.
There are two general directions for future research.
The theoretical direction is to examine the probabilistic representation in other popular DNNs, e.g., the residual networks, which can further verify the proposed probabilistic representation.
The practical direction is to validate the proposed regularization in more complex DNNs and datasets.
A T HE PARAMETERS OF DNN S CANNOT SATISFY THE I . I . D . PRIOR A. A MLP ON THE MNIST DATASET
We design a MLP (Figure 1) to classify the benchmark MNIST dataset.
The MLP has two hidden layers, each hidden layer is a fully connected layer with the sigmoid activation function, and the output layer is the softmax.
Table 3 summarizes the architecture of the MLP.
After the MLP is well trained (its training and testing accuracy is 97 .
7% and 97 .
1% , respectively), we quantify the correlation of different weights in the same layer based on Equation 22 and derive the correlation matrix for all the weights in every hidden layer.
[Equation]
where α n = 1 M (cid:80) Mm =1 α mn is the sample mean of { α mn } Mm =1 .
To examine if different weights are correlated, we only need to check whether their correlation coefﬁcients are close to zero, so we take the absolute value of the correlation matrix element-wisely.
Figure 8 visualizes the correlation matrices of each hidden layer, i.e., A (cid:48)
N × N , B (cid:48)
K × K , and C (cid:48)
L × L .
We ﬁnd that most weights in f 1 are uncorrelated because their correlation coefﬁcients are very close to zero, i.e., | r ( α n , α n (cid:48) )
| ≈ 0 , but many weights in f 2 and f Y are correlated to others, i.e., | r ( β k , β k (cid:48) )
| (cid:54) = 0 and | r ( γ l , γ l (cid:48) )
| (cid:54) = 0 .
In addition, we derive the average of the absolute correlation coefﬁcients of all the weights in each layer and show them at the top of Figure 8, which further validates that the weights of some hidden layer are correlated.
Therefore, the parameters of the MLP cannot satisfy the i.i.d.
prior.
[Figure]
[Table] [Table]
A. A N EURAL N ETWORK WITH A SINGLE HIDDEN LAYER ON THE MNIST DATASET
Based on the seminal work (Neal, 1994), we design a neural network with a single fully connected hidden layer (Figure 9) for classifying the benchmark MNIST dataset (LeCun et al., 1998).
The input layer dimension is 1 × 784 , the hidden layer has 128 neurons with the sigmoid activation function, and the output layer is the softmax with 1 × 10 dimension.
Table 4 summarizes the architecture of the network.
In addition, { α mn } Mm =1 denote the weights of the n th linear ﬁlter in the fully conntected hidden layer and { β nl } Nn =1 are the weights of the l th linear ﬁlter in the output layer.
Therefore, the hidden layer has 128 linear ﬁlters with 1 × 784 dimension, i.e., { α mn } 784 m =1 .
The output layer has totally 10 linear ﬁlters and their dimension is 1 × 128 ,
i.e., { β nl } 128 n =1 .
[Figure]
After the network is well trained (its training and testing accuracy is 97 .
4% and 96 .
9% , respectively), we calculate the absolute sample correlations, i.e., | r ( α n , α n (cid:48) )
| and | r ( β k , β k (cid:48) ) | , and derive two absolute correlation matrices, i.e., A (cid:48)
N × N and B (cid:48)
L × L , based on Equation 2.
They are shown in
Figure 10.
We ﬁnd that most | r ( α n , α n (cid:48) ) | are close to zero, but | r ( β k , β k (cid:48) )
| not.
That indicates the parameters of the NN do not satisfy the necessary condition, thus they cannot satisfy the i.i.d.
prior.
[Figure]
[Table]
A. A CNN ON THE CIFAR-10 DATASET
We design a CNN to classify the benchmark CIFAR-10 dataset (Krizhevsky, 2009).
The CNN has two convolutional layers without activation function, two max-pooling layers for dimension reduction, and two fully connected layers with the sigmoid activation function.
In addition, the output layer is the softmax.
Table 5 summarizes the architecture of the CNN.
After we train the CNN well (its training and testing accuracy are 98 .
0% and 69 .
4% , respectively).
We quantify the correlation of different weights in the same layer based on Equation 2, thereby deriving the absolute correlation matrix for all the weights in every hidden layer.
Since x has 3 channels and f 1 has 64 channels, f 1 has totally 3 × 64 = 192 ﬁlters.
Hence, the dimension of the correlation matrix for f 1 is 192 × 192 .
In particular, we ﬂatten the output of the second convolutional layer, i.e., reshaping 8 × 8 × 128 to 1 × 8192 , thus the ﬁlter dimension of f 3 is 8192 × 1024 .
Table 5 shows the dimension of the correlation matrices for all the layers in the CNN.
Figure 11 shows the absolute correlation matrices for every layer.
We can ﬁnd that most weights are correlated in f 1 and f 2 , i.e., the weights in the convolutional layers are correlated.
In addition, the weighs in the output layer are also shown to be correlated.
However, the weights in f 3 and f 4 are not correlated, because their correlation coefﬁcients are close to zero.
In other words, the weights in the fully connected layers are uncorrelated.
Moreover, we derive the average of the absolute correlation coefﬁcients of all the weights in each layer and show them at the top of Figure 11, which further validates the correlation of the weights in each layer.
Overall, the parameters of the CNN cannot satisfy the necessary condition.
Therefore, they cannot satisfy the i.i.d.
prior.
[Figure]
A. N EURAL NETWORKS WITH A SINGLE HIDDEN LAYER ON A SYNTHETIC DATASET
In order to further demonstrate that the parameters of DNNs cannot satisfy the i.i.d.
assumption, we construct a neural network, i.e., NN = { x ; f ; f Y } shown in Figure 9, to classify a synthetic dataset.
The synthetic dataset consists of 4,800 32 × 32 grayscale images with 4 classes, and every synthetic image is sampled from the Gaussian distribution N (0 , 1) .
To make the synthetic image have certain features that can be learned by neural networks, we sort the synthetic image in the primary or secondary diagonal directions by the ascending or descending orders.
Thus, the synthetic dataset only has four features, i.e., four classes, and their corresponding images are drawn above.
There are two important reasons why we choose the synthetic dataset.
Above all, compared to benchmark datasets containing too complex features, the synthetic dataset only has four simple features, thus we can easily show the correlation of different parameters by simply visualizing them.
In addition, since a single synthetic image only has a single feature, we can easily examine the connection between the parameters of networks and the synthetic image.
[Figure]
Since the synthetic dataset is very simple, we set the hidden layer f have only eights neurons and the output layer f Y have four output nodes.
After training the network well, we show all the eight neurons in Figure 13.
In contrast to the i.i.d.
assumption, their weights show strong correlation in three aspects.
First, weights show strong internal correlation within each neuron, i.e., if a weight has large magnitude, its neighbors have high probability to be large, and vice versa.
Second, weights show strong external correlation between different neurons.
For example, most weights of Neuron and Neuron5 at the same position show strong positive correlation, and most weights of Neuron and Neuron7 at the same position show strong negative correlation.
Third, the weights show strong dependence on the training dataset.
More speciﬁcally, since all the features of the synthetic dataset concentrate in the primary and secondary diagonal directions, most linear ﬁlters also demonstrate related features in the same directions.
[Figure]
The correlation
[Figure] [Figure]
[Figure]
Figure 14 visualizes the four linear ﬁlters in the output layer.
Though they not show too strong internal correlation because their dimension is too small, their weights at the same position still show strong external correlation, e.g., the weights of Filter0 and
Filter1 at the same position show strong negative correlation.
Moreover, we calculate the correlation coefﬁcients of all the four ﬁlters based on Equation 2 and derive the correlation matrix show in Figure 14.
It shows that the correlation coefﬁcient of Filter0 and Filter1 is close to − 1 , which further validates that they have strong negative correlation, hence the parameters of the output layer do not satisfy the i.i.d.
prior.
Compared to the simulations using benchmark datasets, the synthetic dataset enable us to expose the correlation of different weights and their dependence on the training dataset in the more clearly way.
Overall, we demonstrate that the parameters of neural networks are not i.i.d..
B P ROOF : T HE NECESSARY CONDITION FOR THE ACTIVATIONS OF HIDDEN LAYERS BEING INDEPENDENT
𝑓 "" 𝑥 " 𝒙 𝑥 % 𝑥 & 𝑓 "’ 𝑓 "( 𝑓 ) (1) 𝛽 ’" 𝛽 "" 𝛽 (" 𝛼 "’ 𝛼 %’ 𝛼 &’ 𝑓 "/ 𝛽 /" ⋮ ⋮ ⋮ ⋮ 𝒇𝒇 𝒀 𝑓 ) (𝐿)
[Figure]
=1 ,
where g 1 n ( x ) = (cid:80) Mm =1 α mn · x m + b 1 n is the n th linear ﬁlter, α mn is the weight of the edge between x m and f 1 n , b 1 n is the bias, and σ 1 is a non-linear activation function.
Similarly, f 2 = { σ 2 [ g 2 k ( f 1 )] } Kk =1 where g 2 k ( f 1 ) =
(cid:80) Nn =1 β nk · f 1 n + b 2 k .
In addition, since f Y is softmax, f Y
= { 1 Z Y exp [ g yl ( f 2 )] } Ll =1 where g yl ( f 2 ) =
(cid:80)
Kk =1 γ kl · f 2 k + b yl , and the partition function is Z Y =
(cid:80) Ll =1 exp [ g yl ( f 2 )] .
In the context of Bayesian probability, all the weights and the biases in the MLP (Figure 15) have prior distributions, i.e., α mn ∼ P ( A mn ) , β nk ∼ P
( W nk ) , b 1 n ∼ P
( B 1 n ) , and b 2 k ∼ P ( B 2 k ) , thus we regard G 1 n = (cid:80)
Mm =
A mn X m + B 1 n as the random variable of g 1 n ( x ) = (cid:80)
Mm =1 α mn · x m + b 1 n , and G 2 k
= (cid:80) Nn
=1 W nk F 1 n + B 2 k as the random variable of g 2 k ( f 1 ) =
(cid:80) Nn =1 β nk · f 1 n + b 2 k .
Therefore, the random variables of arbitrary activations in f 1 and f 2 , e.g., f 1 n and f 2 k , can be expressed as F 1 n = σ 1 ( G 1 n ) and F 2 k = σ 2 ( G 2 k ) , respectively.
Based on the theorem (Appendix D) that functions of independent random variables are independent, we can derive that if { F 2 k } Kk =1 are independent and the activation function σ 1 ( · ) is invertible, then { G 2 k } Kk =1 are independent.
In other words, in order to prove { F 2 k } Kk =1 are independent, we only need to prove { G 2 k } Kk =1 being independent as long as σ 2 ( · ) is invertible.
The necessary condition for { G 2 k } Kk =1 being independent is ∀ ( k, k (cid:48) )
∈ S 1 = {
[Equation]
k,
[Equation]
) ∈
[Equation]
|
[Equation]
= k (cid:48) , 1 ≤ k ≤ K, 1 ≤ k (cid:48) ≤ K } , the covariance Cov ( G 2 k , G 2 k (cid:48) )
= 0 , which can be formulated as
[Equation]
Based on Cov ( X + Y, W + Z ) =
Cov ( X,
[Equation]
( Y, Z ) , we can derive that
[Equation]
Based on the deﬁnition of covariance
, i
[Equation]
e., Cov ( A, B ) =
E ( AB ) − E
( A ) E ( B ) , we have
[Equation]
Based on the law of total expectation, i.e, E ( AB ) =
E A ( AE B | A ( B )) , we have
[Equation]
[Equation]
Since the statistical properties of the input { F 1 n } Nn =1 are unrelated to the weights β nk and biases b 2 k , we have E F 1 n F 1 n
[Equation]
W nk W
[Equation]
[Equation] [Equation]
1 n )
[Equation]
E F 1 n ( F 1 n ) , and E F 1 n | W nk ( F 1 n ) =
E F 1 n ( F 1 n ) .
As a result
[Equation]
we can derive that
[Equation]
Substituting Equation (27) for the corresponding expectations in Equation (25)
[Equation]
we have
[Equation]
Assuming { F 1 n } Nn =1 are i.i.d.
and F 1 n ∼ P ( F 1 ) , we have E ( F 1 n F 1 n (cid:48) )
=
E ( F 1 n ) E ( F 1 n (cid:48) )
=
E 2 ( F 1 ) .
Hence, we can derive that
[Equation]
Substituting Equation (29) for the corresponding covariances in Equation (24), we have
[Equation]
Based on Cov ( X + Y, W + Z ) =
Cov ( X + W ) + Cov ( X
[Equation]
Z )
[Equation]
Cov
[Equation]
Y
[Equation] [Equation]
Y + Z ) , we can derive that
[Equation]
Therefore, given σ 2 ( · ) is invertible and { F 1 n } Nn =1 are i.i.d., the necessary condition for { F 2 k } Kk = being independent can be expressed as
[Equation]
where W k = (cid:80)
[Equation]
and W
[Equation]
Overall, in order to guarantee the activations of a hidden layer are independent with each other in the context of Bayesian probability, the prior distributions of the weights and biases of the hidden layer should satisfy the above necessary condition given the inputs of the hidden layer are i.i.d.
and the activation functions are invertible.
C P ROOF : T HE NECESSARY CONDITION FOR THE ACTIVATIONS OF HIDDEN LAYERS BEING IDENTICALLY DISTRIBUTED
Given the same DNN drawn in Figure 15, we assume σ 2 ( · ) being strictly increasing and differentiable, so σ 2 ( · ) is invertible and its inverse σ − 1 2 ( · ) is also strictly increasing.
The cumulative distribution function of F 2 k can be expressed as
[Equation]
) where φ ( F 2 k ≤ f ) is the probability of F 2 k takes on a value less than or equal to f .
Subsequently, we can obtain that
[Equation]
Equation (34) means that if the activation function σ 2 ( · ) is strictly increasing and differentiable, { F 2 k } Kk =1 being identically distributed implies { G 2 k } Kk =1 being identically distributed.
In the context of Bayesian probability, G 2 k = (cid:80)
Nn =1 W nk F 1 n + B 2 k , we can derive that
[Equation]
Based on the law of total expectation and Equation 27, we have
[Equation]
Assuming { F 1 n } Nn =1 are i.i.d.
and F 1 n ∼ F 1 , we have E ( F 1 n ) =
E ( F 1 ) .
Hence, we can derive that
[Equation]
Based on the property of expectation, i.e., E ( A ) + E ( B ) =
E ( A + B ) , we have
[Equation]
Therefore, given σ 2 ( · ) is strictly increasing and differentiable and { F 1 n } Nn =1 are i.i.d., the necessary condition for { F 2 k } Kk =1 being identically distributed is that ∀ ( n, n (cid:48) )
∈ S 1 , we have
[Equation]
where W k = (cid:80) Nn =1 W nk and W k (cid:48)
= (cid:80) Nn =1 W nk (cid:48) .
Overall, in order to guarantee the activations of a hidden layer are identically distributed, the prior distributions of the weights and the biases of the hidden layer should satisfy the above necessary condition in the context of Bayesian probability given the inputs of the hidden layer are i.i.d.
and the activation functions are strictly increasing and differentiable.
D T HEOREM : F UNCTIONS OF INDEPENDENT RANDOM VARIABLES ARE INDEPENDENT
Let X and Y be independent random variables on a probability space (Ω , F , P ) .
Let g and h be real-valued functions deﬁned on the codomains of X and Y , respectively.
Then g ( X ) and h ( Y ) are independent random variables.
Proof: Let A ⊆ R and B ⊆ R be the range of g and h , the joint distribution between g ( X ) and h ( Y ) can be formulate as P ( g ( X )
∈
A , h ( Y ) ∈ B ) .
Let g − 1 ( A ) and h − 1 ( B ) denote the preimages of A and B , respectively, we have
[Equation]
[Equation]
Based on the deﬁnition of independence, we can derive that
[Equation]
Based on the deﬁnition of preimage, we can derive that
[Equation]
Therefore, g ( X ) and h ( Y ) are independent random variables.
E T HE NECESSARY CONDITIONS FOR ACTIVATIONS BEING I . I . D . ARE VALID FOR CONVOLUTIONAL LAYERS
Assuming a DNN = { x ; f 1 ; ...
; f I ; f Y } has two adjacent convolutional layers f i and f i +1 .
The ﬁrst convolutional layer has N channels, i.e. f i = { f ni } Nn =1 , and the second convolutional layer has
K channels, i.e. f i +1 =
{ f ki +1 } Kk =1 .
Therefore, the dimension of the convolutional ﬁlters
S i +1 in f i +
are W × H × N × K , i.e., the dimension of a single ﬁlter connecting the f ni channel and the f k i +1 channel are D ( S ( k,n ) i +1 ) =
W × H .
As a result, the f k i +1 channel can be formulated as
[Equation]
(43) where σ i +1 is the activation function and b i +1 is the bias for the ( i + 1) th channel.
A B
C D 𝑥 " 𝑥 # 𝑥 $ 𝑥 % 𝑥 & 𝑥 ’ 𝑥 ( 𝑥 ) 𝑥 *
A B
C D
A B
C D
A B
C D
A B
C D 𝑥 " 𝑥 # ⋮ 𝑥 ) 𝑥 * ∘ 𝑦 "
𝑦 # 𝑦 $ 𝑦 %
A convolution channel
A fully connected layer
Convolutional filters
Input
Output 𝑦 " =
𝐴 0 𝑥 " + 𝐵 0 𝑥
# + 𝐶
𝑥 % + 𝐷 0 𝑥 & 𝑦 # =
𝐴 0 𝑥
+ 𝐵 0 𝑥
$ + 𝐶 0 𝑥 & + 𝐷 0 𝑥 ’ 𝑦 $ =
𝐴
𝑥 %
+ 𝐵 0 𝑥 & + 𝐶 0 𝑥 ( + 𝐷 0 𝑥 ) 𝑦 %
=
𝐴 0 𝑥 & + 𝐵 0 𝑥 ’ + 𝐶 0 𝑥 )
+ 𝐷 0 𝑥 * = 𝑏 𝑥6 " 𝑥6 # 𝑥6 $ 𝑥6 % 𝑥6 & 𝑥6 ’ 𝑥6 ( 𝑥6 ) 𝑥6 *
E F
G H ∘ 𝑦6 " 𝑦6 # 𝑦6 $ 𝑦6 % = 𝑧 " 𝑧 # 𝑧 $ 𝑧 %
+ +
E F
G H
E F
G H
E F
G H
E F
G H 𝑥6 " 𝑥6 # ⋮ 𝑥6 ) 𝑥6 * 𝑏 + = 𝑧 " 𝑧 # 𝑧 $ 𝑧 % =
[Figure]
is the
[Figure]
𝒙
Since Garriga-Alonso et al.
(2018) demonstrate the equivalence between a convolution operation and the matrix multiplication, we can regard the convolutional channel f ki +1 as a fully connected layer.
Speciﬁcally, the input of the virtual fully connected layer is the same as the input of the convolutional channel, the output of the virtual fully connected layer is the vectorized output of the convolutional channel, and the weights of the virtual fully connected layer depend on the convolutional ﬁlter and the spatial location of the output of the convolutional channel.
Without considering the activation function, the equivalence between a convolutional channel and a fully connected layer can be explained in Figure 16, in which the ﬁrst convolutional layer has two channels, i.e., f i = { x ; ˆ x } , where x = { x 1 , · · · , x 9 } and ˆ x = { ˆ x 1 , · · · , ˆ x 9 } , and the k th channel of the second convolutional layer is f ki +
=
{ z 1 , z 2 , z 3 , z 4 } .
In addition, the dimension of the convolutional ﬁlter S i +1 is 2 × 2 × 2 × 1 , where S ( k, 1) i +
=
[ A, B ; C, D ] and S ( k, 2)
i +1 =
[ E, F ; G, H ] .
Figure 16 shows that the output of f ki +1 can be expressed as a fully connected layer.
The input of the fully connected layer is the vector x , the weights of each neuron is represented by each row of the matrix W , and the bias is b , thus f k i +
= W · x + b · 1 .
If we consider the activation function and make a precise statement, f ki +1 can be formulated as
[Equation]
(44) where W ki +1 is the matrix corresponding to all the convolutional ﬁlters in the k th channel.
To derive the necessary conditions for two arbitrary different channels are i.i.d.
, we need to derive the necessary conditions for two arbitrary activations in two arbitrary different channels are i.i.d.
For example, if the m th activation of the k channel and the m (cid:48) th activation of the k (cid:48) channel in f i +1 are i.i.d., the necessary conditions are Cov ( f ki +1 ( m ) , f k (cid:48)
i +1 ( m (cid:48) ))
= 0 , E ( f ki +1 ( m ))
=
E ( f k (cid:48)
i +1 ( m (cid:48) )) .
Since we demonstrate the equivalence between a convolutional channel and a fully connected layer, the necessary conditions for activations being i.i.d.
in a fully connected layer are also valid for a convolutional layer.
In other words, we can also use the Equation 3, Equation 4, and Equation 5 to examine whether activations being i.i.d.
is valid in a convolutional layer under the assumption that all activations being i.i.d.
in the previous layer.
[Table]
F A CTIVATIONS BEING I . I . D . CANNOT BE VALID FOR ALL THE HIDDEN LAYERS OF DNN S
In this section, we demonstrate that activations being i.i.d.
cannot be valid for all the hidden layers of DNNs through showing that the hidden layers of two typical DNNs, i.e, the MLP and the CNN, cannot satisfy the necessary conditions for activations being i.i.d..
F. T HE MLP ON THE MNIST DATASET
The architecture of the MLP for classifying the MNIST dataset is visualized in Figure 15, and its speciﬁc parameters are summarized in Table 3.
Since the activation functions of all the hidden layers in the MLP are the sigmoid function, which is strictly increasing, differentiable, and invertible, we only need to examine the ﬁrst two necessary conditions in the MLP.
Since we restrict all the hidden layers from using the bias, the ﬁrst two necessary conditions for activations being i.i.d.
can be simpliﬁed as
[Equation]
2 | k
[Equation]
=
[Equation]
≤
[Equation]
≤ K
[Equation]
≤ k
[Equation]
≤ K } , we must have
[Equation]
Based on the architecture of the MLP, Table 6 summarizes all the statistical measures for each layer of the MLP.
Therefore, the key to examine if the layers of the MLP satisfy the necessary conditions is to estimate the statistical measures.
To estimate the expectations E ( X ) , E ( F 1 ) , and E ( F 2 ) , we use the sample means x , f 1 , and f 2 , respectively.
The deﬁnition of the sample mean is
[Equation]
After the MLP is well-trained, we derive the sample means as x = 0 .
131 , f 1 = 0 .
436 , and f 2 = 0 .
498 given the benchmark MNIST testing dataset.
Taking into account the estimation error, we can regard E ( X )
= 0 .
As a result, we can derive that ∀ ( n, n (cid:48) ) ,
Cov ( G 1 n , G 1 n (cid:48) )
=
and E ( G 1 n ) =
E ( G 1 n (cid:48) )
= 0 , thus f 1 satisfy the necessary conditions for activations being i.i.d..
However, f 1 = 0 .
436 and f 2 = 0 .
498 imply that E ( F 1 ) (cid:54) = 0 and E ( F 2 ) (cid:54) = 0 even considering the estimation error.
Therefore, we need to estimate Cov ( W k , W k (cid:48) ) and E ( W k ) to examine whether the activations { f 2 k } Kk =1 being i.i.d..
Since the training dataset is commonly viewed as i.i.d., we can use the same training dataset to train the MLP several times to derive multiple independent observation samples of { β nk ( t ) }
Tt =1 to estimate the random variable W nk , thereby estimating Cov ( W k , W k (cid:48) ) and E ( W k ) .
[Figure]
n
E ( A n ) f 1 (6.75)   60 k 1.
E ( W k ) f 2 (0.41)    8 l
E ( C l ) f Y (4.06)
More speciﬁcally, we use the sample correlation r ( β k , β k (cid:48) ) to check whether Cov ( W k , W k (cid:48) ) are close to zero and use the sample mean β k to estimate E ( W k ) .
[Equation]
where β k = 1 T (cid:80)
Tt =
1 β k ( t ) is the sample mean of { β k ( t ) }
Tt =1 and β k ( t ) =
(cid:80) Nn =1 β
nk ( t ) .
In order to make a tradeoff between the estimation precision and computation complexity, we use the same training dataset to train the MLP 20 times to derive the samples, i.e., { β nk ( t ) } 20 t =1 .
For an activation f 2 k , it has 128 weights because its previous layer has 128 neurons.
Hence, a single sample β k ( t )
=
(cid:80) 128 n =1 β nk ( t ) .
Based on the 20 samples, i.e., { β k ( t ) } 20 t =1 , we can calculate r ( β k , β k (cid:48) ) to examine whether Cov ( W k , W k (cid:48) ) are close to zero and derive the sample mean to estimate E ( W k ) .
We use the same method to estimate Cov ( A n , A n (cid:48) ) , E ( A n ) , Cov ( C l , C l (cid:48) ) , and E ( C l ) .
Finally, we shows the absolute correlation matrices i.e., A N × N , W K × K , and C L × L , in Figure 3.
Since many elements in the three matrices are far from zero, we conclude that their corresponding covariances, i.e, Cov ( A n , A n (cid:48) ) ,
Cov ( W k , W k (cid:48) ) , and Cov ( C k , C k (cid:48) ) , are not close to zero based on connection between correlation and covariance
[Equation]
where σ W k > 0 is the standard derivation of W k .
Figure 17 visualizes the expectations, i.e., E ( A n ) , E ( W k ) , and E ( C l ) , for each layer in the MLP.
We ﬁnd that some expectations show great variations, especially { E ( A n ) } Nn =1 and { E ( C l ) } Ll =1 .
In other words, the MLP cannot guarantee that all the expectations of the weights summation are equivalent even though we take into account the estimation error.
Therefore, the MLP cannot satisfy the necessary condition for activations being identically distributed.
Overall, we conclude that not all the hidden layers of the MLP can satisfy the necessary conditions for activations being i.i.d.
under the assumption that the activations of the previous layer are i.i.d.
F.
T HE CNN ON THE CIFAR-10 DATASET
The architecture of the CNN for classifying the CIFAR-10 dataset is in Table 5.
In particular, the two convolutional layers do not have activations function, i.e., their activation functions can be viewed as y = x , which is strictly increasing, differentiable, and invertible.
The sigmoid activation function is also strictly increasing, differentiable, and invertible, thus we only need to examine the ﬁrst two necessary conditions in the CNN.
To simplify derivation and decrease computation complexity, we assume the max-pooling layers are special activation functions and satisfy the third necessary condition for activations being i.i.d.
[Table]
Similar as the MLP, we also restrict all the layers of the CNN from using the bias, thus the ﬁrst two necessary conditions for activations being i.i.d.
can be simpliﬁed as
[Equation]
} , we must have
[Equation]
Table 7 summarizes all the statistical measures for every layer of the CNN based on the architecture of the CNN in Table 5.
The deﬁnitions of most random variables are similar as the MLP except the dimension of the weights.
To check if two arbitrary convolutional channels are i.i.d.
, we need to know all the weights of two convolutional channels.
Based on the equivalence between a convolutional channel and a fully connected layer (Appendix E), all the weights of a single convolutional channel consist of all the convolutional ﬁlters related to the convolutional channel.
For example, the dimension of all the convolutional ﬁlters related to a convolutional channel in f 1 are 3 × 3 × 3 , thus a single convolutional channel in f 1 has 27 weights, so the weights dimension of f 1 is 27 × 64 .
After training the CNN well (its training accuracy is 96 . 2% and testing accuracy is 67 .
9% ), we use the same methods as the MLP to estimate the above statistical measures in Table 7.
[Equation]
128×
[Equation]
C 1024×
[Equation]
[Figure] [Figure]
f
[Figure] [Figure]
[Figure] [Figure]
l ) f
[Figure] [Figure]
q
[Figure] [Figure]
E ( D q ) f 4 (13.91)
[Figure]
[Figure]
E ( E r ) f Y (1.13)
For the activation expectations of each layer in the CNN, we have x = − 0 .
023 , f 1 = − 707 . 074 , f 2 = − 6347 .
836 , f 3 = 36 . 870 , f 4 = 15 .
131 , thus all the expectations cannot be zero except the input layer, i.e., E ( X ) = 0 , E ( F 1 ) (cid:54) = 0 , E ( F 2 ) (cid:54) = 0 , E ( F 3 ) (cid:54) = 0 , E ( F 4 ) (cid:54) = 0 .
For the covariances, we show all the correlation matrices of each layer of the CNN in Figure 18.
We ﬁnd that many correlation coefﬁcients are far from zero in all the correlation matrices, thus their corresponding covariances are not close zero either.
Based on the estimations for the activation expectation and the covariance between different weights summations in each layer, we can derive that activations being independent cannot be valid for all the layers of the CNN.
For the weights summation expectation, we visualize all the expectations for each layer in the CNN in Figure 19.
We ﬁnd that some expectations have great variations, especially { E ( C l ) } Ll =1 and { E ( D q ) } Qq =1 , which means that most expectations of weights summations of different neurons in f 3 and f 4 cannot be equivalent.
As a result, the activations of the two fully connected layers in the
CNN cannot be identically distributed.
Overall, we conclude that not all the hidden layers of the CNN can satisfy the necessary conditions for activations being i.i.d.
under the assumption that the activations of the previous layer are i.i.d.
G T HE EQUIVALENCE BETWEEN THE G RADIENT DESCENT ALGORITHM AND THE FIRST ORDER APPROXIMATION
In the context of deep learning, most learning algorithms belong to the gradient descent algorithm.
Given a DNN = { x ; f 1 ; ... ; f I ; f Y } and a cost function H [ f Y , P ( Y | X )] , where P ( Y | X ) is the true distribution of the training label given the corresponding training dataset.
Let θ be the parameters of the DNN, the gradient descent aims to optimize θ by minimizing H [ f Y , P ( Y | X )]
(Rumelhart et al., 1986).
We typically update θ iteratively to derive θ ∗ , which can be expressed as
[Equation]
52) where ∇ θ t H [ f Y , P ( Y | X )] is the Jacobian matrix of H [ f Y ,
P
( Y | X )] with respect to θ t at the t th iteration, and α > 0 denotes a constant indicating the learning rate.
Since P ( Y | X ) is constant, we denote H ( f Y ) as H [ f Y , P
( Y | X )] for simplifying the following derivation.
Since the functions of all hidden layers are differentiable and the output of a hidden layer is the input of its next layer, the Jacobian matrix of H ( f Y ) with respect to the parameters of the i th hidden layer, i.e., ∇ θ ( i ) H ( f Y ) , can be expressed as follows based on the chain rule.
[Equation]
where θ ( i ) denote the parameters of the i th hidden layer.
Based on Equation 52 and Equation 53, θ ( i ) can be learned by the gradient descent method as
[Equation]
Table 8 summarizes the backpropagation training procedure for the MLP in Figure 15.
[Table]
If an arbitrary function f is differentiable at point p ∗ in R N and its differential is represented by the
Jacobian matrix ∇ p f , the ﬁrst order approximation of f near the point p can be formulated as
[Equation]
(55) where o ( || p − p ∗ || ) is a quantity that approaches zero much faster than || p − p ∗ || approaches zero.
Based on the equivalence between the gradient descent method and the ﬁrst order approximation (Battiti, 1992), updating the activations of the hidden layers of the MLP in Figure 15 during the backpropagation training procedure can be approximated as
[Equation] [Equation]
where f 2 [ f 1 , θ t (2)] denote the activations of the second hidden layer based on the parameters learned in the t th iteration, i.e., θ t (2) , given the activations of the ﬁrst hidden layer, i.e., f 1 .
The deﬁnitions of f 2 [ f 1 , θ t +1 (2)] and f 1 ( x , θ t (1)) are the same as f 2 [ f 1 , θ t (2)] .
Because f 2 has K neurons, i.e., f 2 = { f 2 k = σ 2 ( (cid:80) Nn =1 β nk · f 1 n + b 2 k ) }
Kk =1 , and each neuron has N + 1 parameters, i.e., θ (2) = { [ β 1 k ; · · · ; β Nk ; b
2 k ] } Kk =1 , the dimension of ∇ θ t (2) f 2 is equal to K × ( N + 1) and ∇ θ t (2) f 2 can be expressed as
[Equation]
57) where ∇ σ 2 f 2 =
∂
f 2 [ f 1 , θ t (2)] ∂σ.
Substituting ( ∇ σ 2 f 2 ) · [ f 1 ; 1] T for ∇ θ t (2) f 2 in Equation 56, we can derive f
[Equation]
If we only consider a single neuron, such as f 2 k , in f 2 , θ t +1 (2 k ) =
[ β 1 k ; · · · ; β Nk ; b 2 k ] and θ t (2 k ) =
[ β ∗ 1 k ; · · · ; β ∗ Nk ; b ∗ 2 k ] , thus [ f 1 ; 1] T · θ t +1 (2 k )
= (cid:80) Nn =1 β nk · f 1 n + b 2 k .
As a result,
Equation 58 with respect to a single neuron can be expressed as
[Equation]
Equation 59 indicates that f 2 k
[ f 1 , θ t +1 (2 k )] can be expressed as its ﬁrst order approximation with an error component based on the activation in the previous iteration, i.e., f 2 k [ f 1 , θ t (2 k )] .
Since ∇ σ 2 f 2 k
= ∂f 2 k
[ f 1 , θ t (2)]
∂σis only related to f 1 and the parameters in the t th training iteration, i.e, θ t (2) , it can be regarded as a constant.
Also note the error component not contains any parameters in the ( t + 1) th training iteration.
In summary, f 2 k ( f 1 , θ t +1 (2 k )) can be reformulated as
[Equation]
60) where C 1 = ∇ σ 2 f 2 k and C 2 = f 2 k ( f 1 , θ t (2 k )) − ( ∇
[Equation]
2 f 2 k )
[Equation]
β
[Equation]
nk
[Equation]
∗ 2 k ] .
Similarly, the activations in the ﬁrst hidden layer, i.e, f 1 , also can be formulated as the ﬁrst order approximation in the context of the gradient descent learning algorithm.
[Figure]
⋮
Output
[Figure] [Figure]
% 𝑓 ) (1) 𝑓 ) (𝐿) ⋮ 𝑓 #(
H T HE PROPOSED PROBABILISTIC REPRESENTATION HOLDS FOR CONVOLUTIONAL LAYERS
This section proves that the proposed probabilistic representation is valid for convolutional layers.
In the above CNN, the input x has Q channels x = { x q } Qq =1 (e.g., Q = 3 if x are color images), fhas N convolutional channels f 1 = { f n 1 } Nn =1 , f 2 has K convolutional channels f 2 = { f k 2 } Kk =1 .
The output layer f Y is the softmax with L nodes, thus its distribution can be formulated as
[Equation]
(61) where Z F Y =
(cid:80) Ll =1 exp ( f yl ) denotes the partition function.
Since { f yl = (cid:80)
[Equation] [Equation]
jl ·
[Equation] [Equation]
b yl } Ll =1 , we can derive
[Equation]
(62) where { f 2 j } Jj =1 is the ﬂattened output of f 2 , γ jl is the weight of the edge between f 2 j and f yl , and b yl denotes the bias.
Since f 2 is a convolutional layer with K convolutional channels, it can be reformulated as { f 2 j } Jj =1 = { f k 2 } Kk =1 .
As a result, P ( F Y ) can be reformulated as
[Equation]
where γ kl is a subset of all the parameters { γ jl } Jj =1 such that (cid:80)
Kk
=1 γ Tkl · f k
=
(cid:80)
Jj =1 γ jl · f 2 j , and γ Tkl is the transpose of γ kl .
Therefore, P ( F Y ) can be reformulated as
[Equation]
where Z (cid:48)
F Y =
Z F Y / [ exp ( b yl )] .
Recall the element-wise matrix power, e.g., exp ( a ) b , where a =
[ a 1 ; a 2 ; a 3 ] and b =
[ b 1 ; b 2 ; b 3 ] , we can derive that exp ( a ) =
[ exp ( a 1 ); exp ( a 2 ); exp ( a 3 )] and exp ( a ) b = [ exp ( a 1 ) b 1 ; exp ( a 2 ) b 2 ; exp ( a 3 ) b 3 ] =
[ exp ( a 1 b 1 ); exp ( a 2 b 2 ); exp ( a 3 b 3 )] .
As a result, exp ( ab ) = exp ( a 1 b 1 + a 2 b 2 + a 3 b 3 ) =
(cid:81) | a | exp ( a ) b , where | a | is the element numbers of a .
Based on the element-wise matrix power, we can reformulate P ( F Y ) as
[Equation]
Moreover, we introduce a new partition function Z F k=
[Equation]
2 exp ( f k 2 ) to guarantee that 1 Z Fk 2 exp ( f k 2 ) is a probability measure.
As a result, P ( F Y ) can be reformulated as
[Equation]
where Z (cid:48)(cid:48) F Y =
Z (cid:48)
F Y / (cid:81)
Kk =
[ Z F k 2 ] γ kl ·| f k 2 | .
Overall, P ( F Y ) can be reformulated as a PoE model, in which each expert is deﬁned by every convolutional channel in f
[Equation]
Since f 2 has K convolutional channels, i.e., f 2 = { f k 2 } Kk =1 , and each channel can be formulated as the summation of all the convolutional channels in f 1 , i.e.,
[Equation]
where S ( k,n )◦ f n 1 is the output of the k th convolutional ﬁlter applying into the n th channel of f 1 , b k 2 is the bias, and σ 2 ( · ) is the activation function.
Therefore, we have
[Equation]
Based on the equivalence between the gradient descent learning and the ﬁrst order approximation (Appendix G), Equation 69 can be approximated as
[Equation]
Let us deﬁne a linear function as ϕ k 2 ( f 1 )
=
(cid:80) Nn =1 S ( k,n )◦ f n 1 + b k 2 · 1 , thus the distribution of the k th convolutional channel in f 2 , i.e., P ( f k 2 ) , can be expressed as
[Equation]
If we regard all the linear ﬁlters { ϕ k 2 ( f 1 ) } K k =1 as potential functions for modeling signal structures of f 1 , we can formulate P ( F 2 ) as a speciﬁc Gibbs distribution, i.e., the Markov Random Fields (MRFs), which can be expressed as
[Equation]
where Z F 2 = (cid:80) f 1 exp [ (cid:80) Kk =1 ϕ k 2 ( f 1 )] is the partition function for P ( F 2 ) .
In summary, we prove that the convolutional layer f 2 formulates a MRF model to describe the statistical property of f 1 .
Here we prove that the convolutional layer f 1 also formulates a MRF model to describe the statistical property of the input x .
Based on Equation 70, we have
[Equation]
where Z (cid:48)
F k 2 =
Z F k 2 / exp ( b k 2 ) .
A B
C D 𝑥 " 𝑥 # 𝑥 $ 𝑥 % 𝑥 & 𝑥 ’ 𝑥 ( 𝑥 ) 𝑥 *
A B
C D
A B
C D
A B
C D
A B
C D 𝑥 " 𝑥 # 𝑥 $ 𝑥 % 𝑥 & 𝑥 ’ 𝑥 ( 𝑥 ) 𝑥 * ∘ ,
A convolution operation
A matrix multiplication
Convolutional filter 𝑺 𝟐(𝒌,𝒏)
Input channel 𝒇 𝟏𝒏
Output channel 𝒇 𝟐𝒌 𝑦 " =
𝐴 , 𝑥 " + 𝐵 , 𝑥 # + 𝐶 , 𝑥 % + 𝐷 , 𝑥 & 𝑦 # = 𝐴 , 𝑥 # + 𝐵 , 𝑥 $ + 𝐶 , 𝑥 & + 𝐷 , 𝑥 ’ 𝑦 $ =
𝐴 , 𝑥 % + 𝐵 , 𝑥 & + 𝐶 , 𝑥 ( + 𝐷 , 𝑥 ) 𝑦 % =
𝐴 , 𝑥 & + 𝐵 , 𝑥 ’ + 𝐶 , 𝑥 ) + 𝐷 , 𝑥 * = =
Blank elements are zeros 𝑦 " 𝑦
# 𝑦 $ 𝑦 % 𝑦 " 𝑦 # 𝑦 $ 𝑦 % 𝑾 𝟐(𝒌,𝒏)
Based on the equivalence between a convolutional channel and a fully connected layer proven in
Appendix E, we can regard a convolutional operation as a matrix multiplication, which is visualized in Figure 21.
As a result, we have
[Equation]
where W ( k,n )is a matrix corresponding to the convolution ﬁlter S ( k,n ).
Based on the element-wise matrix power, P ( f k 2 ) can be reformulated as
[Equation]
Similarly, we introduce a new partition function Z F n= (cid:80) f n 1 exp ( f n 1 ) to guarantee that 1 Z Fn 1 exp ( f n 1 ) is a probability measure.
As a result, P ( f k 2 ) can be reformulated as
[Equation]
where Z (cid:48)(cid:48) F k 2 =
Z (cid:48)
F k 2 / (cid:81)
Nn =1 [ Z F n 1 ] W
[Equation]
k
[Equation]
Therefore, P ( f k 2 ) can be reformulated as a PoE model, in which all the experts are deﬁned as P ( f n
[Equation]
=
[Equation]
Since f 1 has N convolutional channels
[Equation]
Based on the same derivation as Equations 70, 71, 72, P ( F 1 ) also can be expressed as a MRF model
[Equation]
where ϕ n 1 ( x ) = (cid:80) Qq =1 S ( n,q )◦ x q + b n 1 · 1 ) and the partition function Z F 1 = (cid:80) x
exp [ (cid:80) Nn
=1 ϕ n 1 ( x )] .
Overall, the distribution of a convolutional layer can be formulated as a MRF model.
[Table] [Table]
I T HE A PPROACH TO GENERATE THE SYNTHETIC DATASET BASED ON NIST AND THE ARCHITECTURE OF THE CNN
In this section, we present a novel approach to generate a synthetic dataset obeying the Gaussian distribution based on the NIST 1 dataset of handwritten digits by class.
The synthetic dataset has similar characteristics as the benchmark MNIST dataset.
It consists of 20,000 32 × 32 grayscale images in 10 classes (digits from 0 to 9), and each class has 1,000 training and 1,000 testing images.
More speciﬁcally, the approach to generate the synthetic dataset is summarized in Algorithm 1, and
Figure 22 visualize the spatial relation of ˆ x = { ˆ x outside , ˆ x outside-boundary , ˆ x inside-boundary , ˆ x inside } .
[Figure]
[Table]
The architecture of the CNN for classifying the synthetic dataset is summarized in Table 9.
J T HE EQUIVALENCE BETWEEN SGD AND VARIATIONAL INFERENCE J. T HE DISTRIBUTION OF THE OUTPUT LAYER GIVEN THE INPUT IS A G IBBS DISTRIBUTION
In Section 6.1, we demonstrate the entire architecture of a DNN = { x ; f 1 ; ... ; f I ; f Y } corresponds to a joint distribution P ( F Y ; F I .. ; F 1 | X ) .
As a result, the conditional distribution of f Y given x , i.e., P ( F Y | X ) , can be formulated as
[Equation]
Since P ( F Y ; F I .. ; F 1 | X )
=
P ( F Y | F I ) · ... · P ( F i
[Equation]
1 | F i ) · ... · P ( F 1 | X ) and only F 2 depends on F 1 , we can derive
P
[Equation]
Since (cid:82) P ( F 2 | F 1 ) P ( F 1 | X
[Equation]
X
[Equation]
1 = P [ F 2 | f 1 ( X ) ] , where f 1 ( · ) is the deterministic function deﬁned by the corresponding hidden layer, P ( F Y | X ) can be simpliﬁed as
[Equation]
Similarly, since only F 3 depends on F 2 , we can obtain
[Equation]
If we iteratively apply the integral with respect to F 3 until F I , we can obtain
[Equation]
Since the output layer f Y is commonly a fully connected layer.
we can formulate P ( F Y | X ) as a multivariate discrete Gibbs distribution based on Equation 8.
[Equation]
[Equation]
It is important to note that the energy function E DNN ( f Y | x ) = − f Y ( f I ( · · · f 2 ( f 1 ( x )) · · · )) , i.e., E DNN ( · ) is determined by the entire architecture of DNNs.
In addition, since P ( F Y | X ) is a conditional distribution of F Y given X , the partition function Z DNN ( x ) should integrate all possible activations of the output layer f Y to guarantee P ( F Y | X ) being a valid conditional probability, i.e.,
Z DNN ( x ) = (cid:82) exp [ f Y ( f I ( · · · f 2 ( f 1 ( x )) · · · )) ]
d f Y ( f I ( · · · f 2 ( f 1 ( x )) · · · )) .
For clarity, here we only consider integral no matter a Gibbs distribution is discrete or continuous, because (cid:80) f 1 ··· f I exp [ − E DNN ( f Y | x )] is a special case of integral.
All functions involving f i should include parameters θ f i in the conditioning set, but we omit for the same reason.
J. SGD CAN BE EXPLAINED AS A VARIATIONAL INFERENCE ALGORITHM
As a dominant paradigm for Bayesian posterior inference P ( H | E ) ∝ P ( E | H ) P ( H ) , the variational inference converts the inference problem into an optimization problem, where the prior distribution
P ( H ) is the probability of arbitrary hypothesis H with respect to the observation E , and P ( E | H ) is the likelihood distribution.
Speciﬁcally, the variational inference posits a family of approximate distributions Q and aims to ﬁnd a distribution Q ∗ ( H ) that minimizes the Kullback-Leibler (KL) divergence between P ( H | E ) and Q ( H ) (Blei et al., 2017).
[Equation]
Notably, recent works demonstrate that SGD performs variational inference during training DNNs (Mandt et al., 2017; Chaudhari & Soatto, 2018).
Following these works, we further conﬁrm the
Bayesian explanation for SGD based on the proposed probabilistic for DNNs.
Given a DNN = { x ; f 1 ; ... ; f I ; f Y } , we choose the cross entropy H
[ P ( Y | X ) , P ( F Y | X )] as the loss function, where P ( Y | X ) is the true posterior distribution determined by the training dataset D and P ( F Y | X ) is the conditional distribution of f Y given x derived from the DNN.
[Equation]
where H [ P ( Y | X )] is the entropy of P ( Y | X ) and P ( y ( l ) | x ) is the true posterior distribution given the training data x .
For example, if L = 4 and P ( y | x ) is { 0 , 1 , 0 , 0 } , so P [ y (2) | x ] =and P [ y (3) | x ] = 0 .
P ( f Y | x ) is the learned conditional distribution of f Y given the same x , e.g.,
P ( f Y | x ) could be { 0 . 2 , 0 . 7 , 0 . 1 , 0 } .
In the context of deep learning, SGD aims to ﬁnd the P ∗ ( F Y | X ) such that it can minimize the loss function H [ P ( Y | X ) , P ( F Y | X )] given the training dataset D .
[Equation]
[Equation]
Given the training dataset D , P ( Y | X ) is a constant, i.e., H [ P ( Y | X )]
= 0 , thus we can obtain
[Equation] [Equation]
Comparing the Equation 88 and 85, we can conﬁrm the equivalence between SGD and variational inference based on the proposed probabilistic explanations for DNNs.
More speciﬁcally, the DNN deﬁnes a family of conditional Gibbs distribution P ( F Y | X ) (Equation 84) and SGD aims to ﬁnd
P ∗ ( F Y | X ) such that it can minimize the distance to the true posterior distribution P
( Y | X ) .
J. SGD IS EQUIVALENT TO THE ENERGY MINIMIZATION OPTIMIZATION
In most deep learning applications, P [ y | x ] is commonly an one-hot vector, i.e., P [ y | x ] only has a single non-zero element.
If we assume that l ∗ is the index of the non-zero element of P [ y | x ] , i.e.,
P [ y ( l ∗ )
| x ] = 1 , we can simplify KL [ P ( Y | X ) (cid:107) P ( F Y | X )]
= − log
[ P ( f Y ( l ∗ ) | x )] .
Based on the deﬁnition of P ( F Y | X ) (Equation 84), we can obtain
[Equation]
where f yl ∗ is the l ∗ th output node of the output layer f Y .
In particular, since log [ Z DNN ( x )] is not related to the output f Y , it can be viewed as a constant with respect to the gradient calculation indicated by Table 8.
Finally, combining Equation 88 and 89, we can obtain
[Equation]
[Equation]
The above equation indicates that SGD is equivalent to the energy minimization optimization, which in turn validates the Gibbs explanation for the hidden layer of DNNs.
[Figure]
K E XPERIMENTS VALIDATING THE THEORETICAL EXPLANATION FOR SGD K. E XPERIMENT : SGD IS EQUIVALENT TO THE ENERGY MINIMIZATION OPTIMIZATION
To validate the proof that SGD learns a P ( F Y | X ) with the minimal energy from the family of
Gibbs distribution determined by a DNN, we use SGD to train the CNN deﬁned in Table 9 for classifying the synthetic dataset shown in Figure 5.
Speciﬁcally, we use the entire training dataset to train the CNN by 30 epochs.
During each training epoch, we calculate the energy function of
P ( F Y | X ) given a testing dataset with the same labels.
Since the testing dataset has the same label, we have the same index of the non-zero element of P [ y | x ] .
As a result, we can easily examine if the corresponding energy function E CNN ( f yl ∗ | x ) is decreasing during the training procedure.
Based on the architecture of CNN in Table 9, E CNN ( f yl ∗ | x ) can be formulated as
[Equation]
In Figure 23, the ﬁrst row visualizes the variation of E CNN ( f yl ∗ | x ) over 30 training epochs.
Though there are certain ﬂuctuations, E CNN ( f yl ∗ | x ) indeed shows a decreasing trend.
Due to limited space, we only visualize the energy minimization procedure given three labels, but the procedure is valid for any label.
In addition, we demonstrate the equivalence on the MLP deﬁned in Table 3 for classifying the MNIST dataset.
The energy function of P ( F Y | X ) corresponding to the MLP is deﬁned as
[Equation]
In Figure 23, the second row shows the same energy decreasing trend during training the MLP.
Overall, we demonstrate the equivalence between SGD and the energy minimization optimization, which in turn validates the Gibbs explanation for DNNs.
K. E XPERIMENT : T HE PARAMETERS LEARNED BY SGD ARE NOT I . I . D .
To validate the parameters learned by SGD not i.i.d.
, we use SGD to train the MLP deﬁned in
Table 3 and the CNN deﬁned in Table 5.
Following the commonly used initialization approach, we initialize all the weights of the two DNNs by Gaussian random numbers, i.e., all the parameters can be viewed as i.i.d.
before training.
We use the sample correlation deﬁned by Equation 2 in Section 4.1 to quantify the dependence between different weights during the training procedure.
Figure 24 visualizes the the variation of the average correlation between different weights of the three layers in the MLP.
Initially, the average correlation for f 1 and f 2 are very small, because parameters are Gaussian random values.
As the training procedure continues, we ﬁnd that all the correlation values are increasing.
Notably, the increase for f 2 is huge, i.e., the weights of f 2 cannot be i.i.d.
anymore.
However, the increase for f 1 is very small, i.e., all the weights of f 1 still could be i.i.d..
Therefore, not all the learned parameters in the MLP are i.i.d..
[Figure]
Figure 25 visualizes the variation of the average correlation between different weights of the ﬁve layers in the CNN.
Similar as the weights in the MLP, the average correlation for f 1 and f 2 initially are very small, because they are initialized as Gaussian random values.
As the training procedure continues, they have a sharp increasing and quickly achieve stationary, thus the parameters of f 1 and f 2 are not i.i.d.
anymore.
In addition, the average correlation for f 3 and f 4 also have an increasing trend, but their increase are very small, i.e., the parameters of f 3 and f 4 still could be i.i
[Figure]
d.. Overall, we conclude that not all the learned parameters in the CNN are i.i.d..
[Figure]
rr
[Figure]
l
[Figure]
t i o n
[Figure]
K. E XPERIMENT : SGD CAN LEARN AN EXPLICITLY B AYESIAN REGULARIZATION
In this section, we demonstrate that SGD learns a prior distribution to regularize the entire DNN.
We design three CNNs = { x ; f 1 ; f 2 ; f 3 ; f 4 ; f 5 ; f Y } in Table 10 to classify the synthetic dataset shown in Figure 5.
We use the entire training dataset to train the CNNs by 30 epochs and use the entire testing dataset to examine if SGD learn a prior distribution to regularize the entire DNN.
Based on the MRF explanation for convolutional layers, the ﬁrst convolutional layer f 1 formulates a MRF model as the prior distribution P ( F 1 ) , which can be expressed as
[Equation]
(93) where ϕ k 1 ( x ) =
S k ◦ x + b k 1 · 1 is the k th linear convolutional channel, S k is the linear convolutional ﬁlter and b k 1 is the bias.
[Table]
Since the synthetic dataset obeys the Gaussian distribution P ( X )
[Table]
we can quantify the regularization learning by averaging KLD [ P ( X ) || P ( F 1 )] over all the testing images for every training epoch.
In addition
[Table]
we use the testing error to measure the generalization performance.
Also note that the only difference between the tree CNNs is they have different K numbers of convolutional channels in f 1 , i.e., different CNNs have different prior distributions.
Therefore, we can derive a comprehensive conclusion of prior distributions learning based on the three CNNs.
Figure 26 shows the prior distribution learning procedure and the corresponding testing error overtraining epochs.
We can ﬁnd that there is a strong positive correlation between KLD [ P ( X ) || P ( F 1 )] and the testing error.
Speciﬁcally, as the learned prior distribution P ( F 1 ) approaching to the truly prior distribution P ( X ) , the testing error decreases as well and when P ( F 1 ) achieves stationary, the testing error also becomes stable on all the three CNNs.
Based on Bayesian regularization theory, we conclude that SGD indeed learns a powerful prior distribution to regularize DNNs to guarantee the generalization performance.
Moreover, we ﬁnd that the CNN with more convolutional channels achieves the smaller testing error, i.e, the better generalization performance, rather than leading to overﬁtting.
In terms of Bayesian theory, that is because a convolutional layer with more convolutional channels formulates a more powerful prior distribution, such that it can convey more prior knowledge of the dataset, thereby rendering better generalization performance, which cannot be clariﬁed by the existing implicit regularization.
[Figure]
L A REGULARIZATION ALGORITHM BASED ON THE PROBABILISTIC REPRESENTATION L. T HE PROPOSED REGULARIZATION ALGORITHM
Based on the probabilistic representation, the ﬁrst hidden layer formulates a MRF model as a prior distribution of the training dataset and other hidden layers recursively serve as prior distributions for their respective next layer in a CNN.
However, the backpropagation calculates the gradient in the backward direction, i.e., updating the gradient of a hidden layer corresponding to a prior distribution must go over all hidden layers behind it.
In addition, the degradation problem could makes the learned prior distribution not precise enough to regularize DNNs and results in overﬁtting, especially in very deep neural networks (He et al., 2016).
In this section, we propose to a new regularization learning algorithm for DNNs, which includes two steps: (i) we learn the prior distributions of DNNs directly from the training dataset { x j } Jj =1 as the blue arrow shown in Figure 27 and (ii) we use the backpropagation to train the initialized DNNs by the learned prior distributions as the red arrow shown in Figure 27.
Since directly learning prior distribution from { x j } Jj =1 is less complicated than the backpropagation learning algorithm, we can relieve the effects of degradation problem and derive prior distributions more precisely, thereby achieving the better generalization performance of DNNs.
An important question need to answer is how many hidden layers of DNNs need to be pre-trained in the ﬁrst step.
Since most prior information is stored in the training dataset { x j } Jj =1 , the hidden layer closest to the input of DNNs, i.e., the ﬁrst hidden layer f 1 , should learn the most prior information from { x j } Jj =1 as long as f 1 has powerful representation ability through including sufﬁcient hidden units, e.g., convolutional ﬁlters.
Therefore, the proposed regularization method only pre-train the ﬁrst hidden layer f 1 in order to decrease the computation complexity.
Pre-training more hidden layers could obtain better generalization performance, which can be a direction for future research.
We choose the Field of Experts (FoE) model to directly learn a prior distribution of { x j } Jj =1 because of two reasons.
Above all, FoE achieves great performance on learning prior knowledge of given datasets on various applications, e.g., image denoising (Roth & Black, 2005), image restoration
Roth & Black (2008), and image inpainting (Schmidt et al., 2011).
More importantly, FoE can be regarded as a Convolutional Neural Networks (CNN) with a single convolutional layer, hence we can easily use the learned FoE model to initialize the ﬁrst hidden layer of CNNs as long as they have the same architecture.
A FoE model can be formulated as
[Equation]
where f k ( · ) is a linear ﬁlter, f expert k is an expert function, and Z FoE = (cid:82) x
(cid:81)
Kk =1 { f expert k [ f k ( x )] } d x is the partition function.
We use the contrastive divergence learning method to train p FoE ( x ) under the
Kullback-Leibler divergence (KLD) criterion (Carreira-Perpinan & Hinton, 2005).
More detail of the training procedure is in (Roth & Black, 2005; 2008).
After deriving a well-trained FoE model, we use the learned linear ﬁlters { f k } Kk =1 to initialize the ﬁrst convolutional layer of CNNs.
Finally, we use the backpropagation algorithm to train the initialized CNN for a speciﬁc application, such as image recognition.
We summarize the proposed regularization learning algorithm in Algorithm 2.
[Equation]
} Jj =1 in the forward direction and calculate the gradient of parameters with respect to the training labels Y for updating the parameters in the backward direction.
L.
S IMULATIONS
In this section, we validate the proposed regularization learning algorithm for image recognition application on the CIFAR-10 benchmark dataset, which includes 70,000 32 × 32 color images in 10 classes, and each class has 6,000 training images and 1,000 testing images.
Speciﬁcally, we randomly choose 2,000 training images of each class as a new training dataset and use the same testing dataset.
Smaller training dataset means overﬁtting being more likely to occur, which can help us validate the proposed regularization learning method more convincingly.
In addition, we covert color images to grayscale to simplify computation.
In order to directly learn regularization from { x j } 20000 j =1 , we design a FoE model with 12 Gaussian
Scale Mixture (GSM) experts, thus it can be expressed as
[Equation]
[Equation]
) where f GSM k [ f k ( x )]
= (cid:80)
[Equation]
=
[Equation]
ki · N ( f k ( x ); 0 , σ 2 b δ ( i ) ) , σ 2 b is a ﬁxed base variance, δ = { δ (1) , · · · , δ (11) } is a range of scales, f k ( · ) is a 3 × 3 convolutional ﬁlter, and π ki denotes the weight of each Gaussian distribution.
The parameters of the FoE model is θ = { f, π } , where f = [ f 1 , ..., f 12 ] and π = { π ki } k =12 ,i =11 k =1 ,i =1 .
The architecture of the CNN for image recognition is summarized in Table 11.
It is noteworthy that the number of convolutional ﬁlters in f 1 and the ﬁlter dimension are the same as the above FoE model, thus we can use the well-trained FoE to initialize f 1 as a regularization for the CNN classifying the CIFAR-10 dataset based on Algorithm 2.
In order to make a comprehensive comparison
[Table]
we choose two commonly used regularizations
[Table]
i
[Table]
e., the dropout (drop rate is 0.3) and the L 2 norm, as references.
We use the testing error to exam the generalization performances of different regularizations.
Figure 28 visualizes that the proposed regularization method (abbr.
Bayes Prior (BP)) successfully decreases the testing error.
In particular, it outperforms dropout ( 32 . 26% ) and achieve similar performance as L 2 norm ( 29 .
Therefore, this experiment validates the effectiveness of the proposed regularization learning method.
[Table]
[Figure]
Moreover
[Figure]
the proposed regularization method can be easily combined with other regularizations methods to improve the generalization performance further.
We combine two arbitrary aforementioned regularization methods together and examine their generalization performances under the same experimental conditions as above.
Figure 29 shows that the combination of two regularizations together achieves better generalization performance than any single one.
In particular, the combination of BP and L2 norm achieves the lowest testing error ( 28 . 60% ).
Also note that if we combine three regularizations together, we achieve the best generalization performance ( 27 .
This experiment further validate the proposed regularization learning algorithm
[Figure]