Neuron with Steady Response Leads to Better Generalization
Abstract
Regularization can mitigate the generalization gap between training and inference by introducing inductive bias.
Existing works have already proposed various inductive biases from diverse perspectives.
However, none of them explores inductive bias from the perspective of class-dependent response distribution of individual neurons.
In this paper, we conduct a substantial analysis of the characteristics of such distribution.
Based on the analysis results, we articulate the Neuron Steadiness
Hypothesis: the neuron with similar responses to instances of the same class leads to better generalization.
Accordingly, we propose a new regularization method called Neuron Steadiness Regularization (NSR) to reduce neuron intra-class response variance.
Based on the Complexity Measure, we theoretically guarantee the effectiveness of NSR for improving generalization.
We conduct extensive experiments on Multilayer Perceptron, Convolutional Neural Networks, and Graph
Neural Networks with popular benchmark datasets of diverse domains, which show that our Neuron Steadiness Regularization consistently outperforms the vanilla version of models with significant gain and low additional computational overhead.
Introduction
Deep Neural Network (DNN) achieves state-of-the-art results in a wide range of areas and has various applications across industries, including self driving cars [37], virtual assistants [38], intelligent healthcare [32], personalized recommendation [34], etc.
DNN’s success usually relies on a plenty amount of training data.
However, DNN’s generalization is often hampered in many domains where training data is insufficient because data annotation is labor-intensive and expensive.
Regularization is a popular method that helps to improve generalization through introducing inductive bias.
Regularization is one of the key elements of machine learning, particularly of deep learning [8].
Specifically, inductive bias represents assumptions about the model properties other than the consistency of outputs with targets.
There have been tremendous efforts in identifying such desired properties, which results in a series of widely used regularization methods.
For example, [-L
-]L
Regularization [36, 21] penalizes large norms of model weights, which puts constraints on “parameter scale".
L1 regularization improves “sparseness" by rewarding zero weight or neuron response.
Jacobian regularization [43, 13] minimizes the norm of the input-output Jacobian matrix to improve the “smoothness" of the learned mapping function.
Orthogonal regularization [4, 1] enlarges “weight diversity" to reduce the feature redundancy.
Batch Normalization [15] promotes “training dynamics stability" by reducing the internal covariate shift.
Although the existing works have already proposed various inductive biases from diverse perspectives, including the aforementioned “parameter scale", “sparseness", “smoothness", “weight diversity" and “training dynamics stability", to the best of our knowledge, there is no work to explore inductive bias from the perspective regarding the characteristics of neuron response distribution on each class.
From another point of view, the existing works leverage the information related to weights (parameter scale regularization), weight correlations (orthogonal regularization), derivatives of mapping function (smoothness regularization), collective neurons responses (sparseness regularization, Batch
Normalization), but none of them considers the intra-class response distribution of individual neurons.
[Figure] [Figure]
In this paper
[Figure]
we study the characteristics of the intra-class response distribution of each individual neuron to identify the new regularization method.
In more detail, for each individual neuron, we analyze the variance of its response to samples of the same class, which is called neuron intra-class response variance .
We find that such intra-class response variance has an obvious correlation with classification correctness.
As shown in Figure [Figure] we find the correctly classified samples usually have smaller intra-class response variance compared to the misclassified samples.
Besides, it can be observed that the vanilla model with cross entropy as the optimization target usually could not control intra-class response variance well, as shown in Figure 2, which leaves a potential improvement space for the regularization.
The details of experiments and observations are explained in Section 2.
Based on these observations, we articulate a Neuron Steadiness Hypothesis : neuron with similar responses to instances of the same class, i.e., smaller neuron intra-class response variance can lead to better generalization.
Accordingly, we propose the regularization method called Neuron Steadiness
Regularization (NSR) to improve generalization by penalizing large neuron intra-class response variance.
Based on the Complexity Measure theory in Section 3.3, we conduct theoretical analysis of the effectiveness of NSR for improving generalization.
In addition, our regularization method shows significant improvements in various network architectures, including Graph Neural Networks (GNN),
Convolution Neural Networks (CNN), and Multilayer Perceptron (MLP).
To sum up, our contributions are as follows:
• We articulate a Neuron Steadiness Hypothesis and demonstrate its validity.
It provides a new regularization perspective based on the neuron-level class-dependent response distribution.
• We propose a new regularization method, Neuron Steadiness Regularization, to improve generalization ability.
The method is computationally efficient and general enough to be applied to various architectures and tasks.
Theoretical analyses guarantee its effectiveness. • Extensive experiments are conducted on multiple types of datasets like images, citation graphs and product graphs, with various network architectures including GNN, CNN and MLP.
Significant improvements evidently verify the effectiveness of Neuron Steadiness Regularization.
Observations
In this section, we verify the Neuron Steadiness Hypothesis by experiments with the following identified observations.
2.1 Correlation between neuron intra-class response variance and classification correctness
The neuron intra-class response variance is derived from neuron response distribution which could be obtained by recording neuron responses when we feed input samples to the model.
In this paper, for the neuron with ReLU as its activation function, we do not take its zero-response into account for calculation, because zero-response corresponds to the inactivated state where the neuron does not respond at all.
In other words, for neurons with ReLU as the activation function, we only record its non-zero responses to represent its response distribution.
With the obtained response distribution of any given neuron, it is relatively straightforward to calculate our discussed statistics, i.e., intra-class response variance, which is the mean squared deviation of the neuron response from the mean of the intra-class response.
Then, for each neuron, we calculate intra-class response variance corresponding to the correctly classified samples and misclassified samples, respectively.
Finally, we aggregate such two respective variances of all neurons in the penultimate layer separately and present them in Figure 1.
Figure 1 shows that for different networks, the average intra-class response variance of correctly classified samples is smaller than that of misclassified ones on arbitrary class.
It indicates the strong correlation between classification correctness and neuron intra-class response variance.
2.2 Dynamics of neuron intra-class response variance during training procedure
We investigate the tendency of neuron intra-class response variance along with the training procedure.
For comparison, we perform the analysis on the vanilla model and the model with our proposed neuron steadiness regularization.
We calculate the intra-class response variance of each neuron on the entire training set after each training epoch.
Then, the intra-class response variance of all neurons is averaged and denoted as average variance in Figure 2 (c).
We also show the testing accuracy and the training cross entropy loss in Figure 2 (a) and (b), respectively.
Other architectures demonstrate similar tendencies and can be found in Appendix D.
From Figure 2, we could see that the cross entropy objective keeps being optimized during the training procedure which leads to increasing classification accuracy for both models.
However, for the vanilla version, the average neuron intra-class response variance is growing larger because the model training does not impose constraints on neuron intra-class response variance.
For the model trained with our proposed regularization, the neuron intra-class response variance is controlled and decreases after a few epochs.
More importantly, the learned model with well-controlled intra-class response variance has higher testing accuracy than the vanilla version although its corresponding cross entropy loss is even larger.
In addition, we could also see that regulating neuron intra-class response variance may also help the optimization procedure to achieve higher accuracy in earlier epochs.
To conclude all the above observations, it is reasonable to design the regularization based on the neuron steadiness hypothesis, i.e., reducing neuron intra-class response variance, for better generalization.
Method
In this section, we first describe the proposed Neuron Steadiness Regularization .
Then we further introduce several techniques for computational efficiency.
3.1 Definition of Neuron Steadiness Regularization
Neuron Steadiness Regularization (NSR) for a specific neuron is defined as the summation of intra-class response variances of different classes.
The NSR for the n -th neuron can be formulated as:
[Equation]
where X n,j is a random variable denoting the n -th neuron’s response for a sample belonging to the j -th class.
J is the number of classes.
α j = z j (cid:80)
i z i is the prior probability of the j -th class where z j is the sample amount of j -th class.
Notice that α j is not a hyper-parameter, and it only presents the importance of different classes.
With the NSR term defined for each individual neuron, the overall regularization is derived by applying NSR term to all neurons in the network as follows:
[Equation]
where L S represents the NSR term for the entire network, and N is the number of neurons in the network. λ n is the hyper-parameter to control the regularization intensity.
In this paper, for practical simplicity, λ is set as the same value for all neurons.
Adding the overall regularization term to the main training target, i.e., the cross entropy loss, the final regularized loss function can be written as:
[Equation]
where L C represents cross entropy loss.
3.2 Practical Implementation 3.2.1 Mini-batch Training
In order to use mini-batch training, we adapt our NSR method to do forward and backward propagation based on mini-batches of samples.
To be specific, we first transform Eq.
(1) as:
[Equation]
The first term has the same form as typical loss functions, i.e., an expectation of the function of data samples, which can be easily generalized to a mini-batch training setting via estimating the expectation with a batch of samples.
Although the second term can be estimated in the same way, the square operation magnifies the estimation error, especially when the batch size is not large enough.
To alleviate the problem while not introducing much computing overhead, we propose a memory queue-based estimation method that allows us to leverage more history samples for estimation without additional sampling and forward/backward computation.
For each class, we record the number of samples and the summation values of the neurons’ response within each batch.
To further reduce the storage overhead, we only maintain these values of the latest M batches by two M -length queues.
More specifically, an element c m,j in the first memory queue is the count number of j -th class instances in the m -th batch, represented as:
[Equation]
where Y m is the set of labels in the m -th batch, y i is the label of i -th sample, and δ is a characteristic function, i.e., δ ( condition )
= 1 if condition is satisfied, otherwise δ ( condition ) = 0 .
An element s ( n ) m,j in the second memory queue is the summation value of n -th neuron’s response for the samples belonging to the j -th class within the m -th batch, represented as:
[Equation]
where X ( n ) m is the set of the n -th neuron’s response within the m -th batch and x ( n ) i is the n -th neuron’s response of the i -th sample.
When a new batch is fed, the estimation of expectation E [ X n,j ] can be updated by the following steps:
[Equation]
where ˆ E [ X n,j ] is the estimation of expectation E [ X n,j ] , C j = (cid:80) m c m,j , S ( n ) j = (cid:80) m s ( n ) m,j , and c ∗ ,j , s ( n ) ∗ ,j , as new elements appended to the queues, represent the count number and the summation for the new batch.
Based on this dynamic update method, the additional memory overhead is negligible, and we give a space complexity analysis in Appendix E.
3.2.2 Layer Selection criterion for Applying NSR
We find that there is a correlation or redundancy among the steadiness constraints of different layers, meaning that applying NSR on different layers of the network has an overlapping effect on neuron steadiness control.
In more detail, from Figure 3, we can see that every layer’s variance ratio decreases even only one specific layer is applied with NSR.
Such steadiness correlation or redundancy among layers is not surprising because different layers are served as input/output of each other.
Due to the aforementioned redundancy, considering the trade-off between performance gain and computational overhead, we select only one layer to apply NSR.
As our NSR is used to reduce neuron intra-class response variance, naturally, the intuitive criterion is to apply NSR to the layer with the largest aggregated neuron intra-class response variance.
Detailed experiment results in RQ4 of the experiment section show such a criterion works well.
In this paper, if not otherwise specified, we apply NSR to only one particular layer determined by this layer selection criterion.
3.3 Theoretical Analysis
In this section, we theoretically analyze the difference in model generalization ability with or without our proposed regularization method NSR based on Complexity Measure [35].
The complexity Measure is one of the mainstream methods to measure the generalization ability of a deep learning model.
A lower complexity measure means a better generalization ability .
Formally, a Complexity Measure is a measure function M : {H , S} → R + where H is a class of models and S is a training set.
According to the definition, if a complexity measure of a given model tends to be 0, the model [Figure] should have the best generalization ability.
Note that, several different complexity measures have been proposed.
We select Consistency of Representations [33] as the concrete Complexity Measure used in our theoretical analysis.
This measure is designed based on Davies-Bouldin Index [5] and is the Winning Solution of the NeurIPS 2020 Competition on Predicting Generalization in Deep Learning.
Mathematically, for a given dataset and a model, we define the following statistics:
[Equation]
where i and j are two different classes, and n i is the number of samples belonging to the class i .
O ( τ ) i is the output representation of the τ -th sample belonging to class i for the final layer, and µ O i is the cluster centroid of the representations of class i .
S i is a measure of scatter within representations of class i , and M i,j is a measure of separation between representations of classes i and j .
Then, the complexity measure based on the Davies-Bouldin Index is defined as:
[Equation]
Based on the definition of the complexity measure C , we have the following lemma:
Lemma 3.1.
For a multi-class classification problem, when (1) a deep learning model utilizing the
Cross Entropy loss with an NSR regularization term on any of its intermediate layers is optimized via gradient descent and (2) the capacity of the model is sufficiently large, the Consistency of Representations Complexity Measure on this model C will tend to be 0.
Under the same condition, when the deep learning model is only optimized by the Cross Entropy loss without an NSR regularization term, there will be infinite local minima where the complexity measure C will be a positive number.
The proof is shown in Appendix A. Intuitively, this lemma shows that in an ideal condition (i.e., the model is sufficiently large), if the definition of the Consistency of Representations Complexity
Measure is utilized, a deep learning model with NSR regularization term will be guaranteed to converge to local minima with the best generalization ability.
In contrast, the model without NSR will not have the same guarantee.
Experiment
In this section, we conduct experiments over a variety of datasets to examine the performance of our neuron steadiness regularization on three extensively used neural network architectures.
We design a series of experiments to answer the following research questions.
RQ1 : How does NSR perform on various datasets and different neural architectures?
RQ2 : Does NSR outperform other classical regularization methods?
RQ3 : What is the effect of combining NSR with other popular methods like Batch Normalization or Dropout?
RQ4 : What if we apply NSR to multiple layers instead of one particular layer?
4.1 Experiment Setup
Network Architectures and Datasets .
Three architectures utilized in experiments are Multilayer
Perceptron (MLP), Convolutional Neural Network (CNN) and Graph Neural Network (GNN).
For vanilla models used as our baselines in different architectures, we adopt ResNet-18 [11], VGG- [26] and ResNet-50 for CNN, GCN [17] and GraphSAGE [10] for GNN.
We run five MLP models which are denoted as MLP-L, L indicates the number of layers including the input layer.
We elaborate on the details of these vanilla models in Appendix C.2.
As for benchmark datasets used in our experiments, MLP and CNN are applied to image recognition task on MNIST [22], CIFAR-10 [19] and ImageNet [6] datasets, respectively.
GNN is applied to node classification on four real-world graph datasets: WikiCS [30], PubMed [48], Amazon-Photo and Amazon-Computers [41].
Notice that ImageNet is a large benchmark dataset with 1000 classes.
Details are shown in Appendix C.4.
[Table] [Table]
Experiment Settings .
We follow the typical implementation settings to conduct our experiments.
For ResNet-18 and VGG19 on CIFAR-10, we follow the detailed setting of [49, 9].
For ResNet-50 on ImageNet, we follow the official implementation provided by torchversion library 4 .
For
GraphSAGE and GCN, we follow the implementation setting of [28].
For MNIST dataset, we divide 60000 training images into the training set with 50000 samples and the validation set with the remaining 10000 samples to select hyper-parameter.
For each of the four graph datasets, it is randomly split into training, validation, and testing sets with a ratio of 6:2:2.
Note that, SGD [40] is used to optimize MLP, and Adam [16] for other models except for ResNet-18 that is optimized by Momentum [40] according to the implementation setting of [49].
We use the typical setting of batch size as 100 for all experiments.
To ensure the model convergence, training epochs are set as 100 for both MLP and GNN, 200 for ResNet and 500 for VGG.
In all experiments except for RQ4, we apply NSR to only one particular layer with the same λ for each neuron.
For this only one hyper-parameter λ , like most other regularization methods, we apply a random search strategy to find its proper value ranging from 1e-2 to 10.
The error rate is the evaluation metric and each result is averaged over 5 runs with different random seeds.
The hardware environments are detailed in Appendix B.
4.2 Experiment Results
RQ1: Performance of NSR : Here, we discuss the performance of NSR over different models and present the results on Tab. 1 - 3 “Gain” is the percentage of relative reduction in the error rate.
Tab. 1 demonstrates that NSR can improve the performance of MLP with different layers: the relative error rate is reduced by 9.39% at least and 43.79% at most.
Besides, as the number of network layers increases, the gain of NSR shows a roughly upward trend.
The four-layer MLP achieves the lowest classification error rate among all vanilla version baselines, and the accuracy becomes worse as the networks grow deeper.
It indicates that deep MLP encounters a severe overfitting problem.
The success of our regularization in addressing such a problem reveals the importance of stabilizing the response of each individual neuron to instances from the same class.
For CNN models, Tab. 2 demonstrates that NSR can reduce the relative error rate for CIFAR-10 by 11.97% on VGG-19 and 11.37% on Resnet-18, and reduce the relative error rate (top-5) for ImageNet by 10.74% on ResNet-50.
It is worth mentioning that VGG-19, ResNet-18 and ResNet-50 have already adopted Batch Normalization, Dropout regularization and Weight Decay in vanilla models, the accuracy gain of our method reveals that NSR can have extra benefits to ulteriorly enhance generalization ability when combined with Batch Normalization, Dropout and Weight Decay.
We will show more evidence about this in RQ3. [Table]
The number of layers of two GNN models varies from 2 to 4.
It is a typical setting following empirical experiences as nodes will capture similar information from neighbors and result in over-smoothness when GNNs grow deeper.
Tab.
3 shows that GraphSAGE and GCN applied with NSR outperform the vanilla model with different layer depths on all datasets.
Specifically, GraphSAGE and GCN achieve an average improvement of 8.6% and 5.8%, respectively, and 17.0% improvement at most.
RQ2: Comparison with Other Regularization : The vanilla ResNet18 here does not use any regularization term and learning rate decay methods for fairness.
We first compare our NSR method with several classical regularization methods shown in Tab. 4.
Notice that, both our NSR and the other regularization methods for comparison have only one hyper-parameter to tune, and we utilize the same hyper-parameter searching strategy to select the best hyper-parameter values for all of them.
As shown in Tab. 4, our NSR performs best among these regularization methods on all three models with different architectures, i.e., MLP-4 (MLP), ResNet-18 (CNN), GraphSAGE (GNN).
The results indicate that our NSR has remarkable improvement and is general for various architectures.
RQ3: Combination with Other Regularization : We further investigate the effect of combining our NSR with other popular methods like Batch Normalization and Dropout.
We conduct the experiments on MLP-4 and the result are organized in Tab. 5.
From Tab. 5 we can find that both
Batch Normalization and Dropout can reduce error rate compared with vanilla baseline, and adding our NSR on top of them can further promote the performance significantly.
It indicates that NSR can provide complementary regularization benefits with Batch Normalization and Dropout. [Table]
[Table] [Table] [Table] RQ4: Layer selection for applying NSR : We have introduced how to select one particular layer to apply NSR in section 3.2.2.
To evaluate such layer selection criterion, we compare the performances trained by applying NSR to multiple layers or to only one selected layer, respectively.
Taking MLPon MNIST and GraphSAGE-2 on Amazon-Computers as two examples, we apply NSR to different layer(s), and their corresponding error rates are listed in Tab.
6 ∼ Tab. 7, where model l means NSR is applied to the l th layer of the model.
Notice that, except the first layer of MLP-4, the input layer, for the second, third, and last layer of MLP-4, their neuron intra-class response variance are 409, 510, and 1660, respectively.
For GraphSAGE-2, its neuron intra-class response variance are 4.15 and 2.68, for the first and last layer, respectively.
The variance of MLP-4 and GraphSAGE-2 are quite different because of the data characteristic difference between MNIST and Amazon-Computers.
The results in Tab. 6 - 7 show that, no matter which layer(s) is applied with NSR, it could always improve the accuracy compared with the vanilla baseline for both MLP-4 and GraphSAGE-2.
In addition, according to our layer selection criterion, applying NSR to the layer with the biggest variance, i.e., the last layer for MLP-4 and the first layer for GraphSAGE-2, could achieve the most significant gain compared with applying NSR to other individual layers.
Also, it could achieve similar accuracy compared with the best one obtained by applying NSR to multiple layers.
This is the empirical evidence to demonstrate the rationality of our layer selection criterion.
Related Work
Regularization improves generalization ability by introducing the inductive bias based on prior knowledge.
According to the type of prior knowledge, existing works can be roughly categorized into Domain Specific Regularization and Model Generic Regularization.
Domain Specific Regularization shows success in various domains including Face Recognition [46, 24, 2, 25], Knowledge Graph Completion [50, 20, 31], Graph Neural Network [3, 14, 39, 47, 27].
In Face Recognition, [2] points out the existence of external factors, such as different situations of environmental illumination, head poses, and facial expressions, which brings the challenge that faces images from the same person may have even larger differences than face images from different persons.
To address specific challenges in face recognition, [24] imposes the carefully-designed regularization method which enforces the representations of face instances from the same person to be similar.
In Graph Neural Network, the popular model usually suffers from the over-smoothing problem due to the six degrees of separation [18].
[3] statistically analyzes the high correlation between smoothness and the mean average distance (MAD) among node representation.
Then a regularization called MADGap is proposed which punishes over smoothness by minimizing MAD.
Model Generic Regularization can be categorized into network-wise Regularization, layer-wise
Regularization, and neuron-wise Regularization, according to the granularity of the studied properties.
Network-wise Regularization encodes the desired property of the entire network, like the sparseness of the network and the smoothness of the mapping function between input and output.
L2 Regularization [36, 21] encourages small sum squared magnitude of model weights.
[13] introduces an efficient framework to minimize the norm of the input-out Jacobian matrix for noise robustness.
[7] encourages model parameters to be uniformly low at some local regions of the loss function.
Layer regularization [45, 36] becomes popular due to the success achieved by Batch Normalization [15].
Neuron-wise
Regularization is the method with the most fine-grained granularity.
Dropout [12, 44] is the typical example belonging to this category.
It randomly removes neurons with a certain probability to avoid strong co-adaption between neurons.
Activation Regularization [29] adds the L2 Regularization on masked activations of neurons and Temporal Activation Regularization restricts the difference between RNN outputs at adjacent timesteps.
To minimize the variance of sample variance and obtain a few distinct modes, Variance Constancy Loss [23] is applied on all neurons before the activation as regularization.
Our proposed Neuron Steadiness Regularization belongs to the neuron-wise regularization that leverages information of individual neuron response distribution.
Conclusion and Future Work
We explore the inductive bias from the new perspective of class-dependent response distribution of individual neurons.
Based on experimental observations, we articulate the Neuron Steadiness
Hypothesis and propose the Neuron Steadiness Regularization that penalizes large intra-class neuron response variance.
Based on the Complexity Measure, we provide a theoretical analysis of its effectiveness for improving generalization.
Additionally, we conduct extensive evaluations on diverse datasets with various network architectures to demonstrate its power.
Especially, we demonstrate its effectiveness on the classification task with large models like ResNet-50, and large datasets with many classes like ImageNet with 1000 classes.
We systematically consider the border impact and No risk is found. Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] (c) Did you discuss any potential negative societal impacts of your work?
[Yes] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] We share the corresponding link in Appendix C (b) Did you specify all the training details (e
g., data splits, hyperparameters, how they were chosen)?
[Yes] See in Appendix C. (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] Yes, each results is calculated from five random seeds: [1, 3, 5, 7, 9] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See in Appendix B 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] (b) Did you mention the license of the assets?
[Yes] See in Appendix C. (c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether
and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
A Proof A.1 Lemma 3.
Lemma 3.1 .
For a multi-class classification problem, when (1) a deep learning model utilizing the
Cross Entropy loss with an NSR regularization term on any of its intermediate layer is optimized via gradient descent and (2) the capacity of the model is sufficiently large, the Consistency of Representations Complexity Measure on this model C will tend to be 0.
Under the same condition, when the deep learning model is only optimized by the Cross Entropy loss without an NSR regularization term, there will be infinite local minima where the complexity measure C will be a positive number.
Proof.
Let us consider the original Cross Entropy loss first:
[Equation]
where N is the number of training samples, y τ is the label of the τ th sample, x τ is the feature of the τ th sample, Θ is the parameters of the model, and q ( y | x τ ; Θ) is the predicted probability modeled by the deep learning model.
Considering a single sample, we have the Cross Entropy loss for the τ th sample:
[Equation]
12) where q τ ( y ) ≜
q ( y =
y
τ
| x τ ; Θ) for brevity.
Because the deep learning models for the classification task always have a normalization operation (e.g., Softmax) to make the final unconstrained representation be a probability form, q τ ( y ) has the following form:
[Equation]
where J is the number of classes, a ( τ ) =
[ a (
[Equation]
) 1 , ..., a ( τ ) J ] is the final representation for the τ -th sample.
The loss can be rewritten as:
[Equation]
[Equation]
where A τ = (cid:80) Jj =1 a ( τ ) j is the normalization factor.
Since we will use gradient descent to optimize the objective function, we calculate the partial derivatives of the objective for the representations h ( τ ) j :
[Equation]
Let the gradient be zero, then we obtain the sufficient conditions of the local minima:
[Equation]
[Equation]
In that case, under the assumption that the model is sufficiently large, the converged local minima for the τ -th sample are the vectors where only the y τ -th entry is an arbitrary positive real number and the other entries are 0.
According to the definition of the Consistency of Representations Complexity Measure Eq.
(10), it is easy to see that we have an infinite number of local minima
where the measures S i and M i,j for arbitrary classes i and j are finite positive numbers.
Thus, the Consistency of Representations
Complexity Measure C will be a positive number in those local minima.
Let us take our proposed NSR regularization term L S (refer to Eq.
(3)) into consideration.
When adding NSR to k -th layer, the loss of NSR can be rewritten as:
[Equation]
where n i is the number of samples belonging to the class i , O ( τ ) i,k is the k -th layer output representation of the τ -th sample belonging to class i , µ O i,k is the cluster centroid of the k -th layer representations of class i .
If we set O ( τ ) i,k as optimization variables, it is easy to see that the global minima is
O ( τ ) i,k
= µ O i,k .
It means that the intermediate representations of the same class in layer k will be the same vector.
Combined with the conditions of the local minima of Cross Entropy loss, we have the optimal final representations of the overall loss, i.e.,
[Equation]
[Equation]
where a τi is the final representation vector of the τ -th sample belonging to the class i , and C i is a positive real number that satisfies C i ̸
=
C j if i ̸
=
j .
It means that the final representations for the samples belonging to the same class are the same, while the representations are different for the samples belonging to different classes.
It is obvious that in that case, S i is zero and M i,j for arbitrary classes i and j is a finite positive number.
Thus, the Consistency of Representations Complexity
Measure C will be 0.
The Lemma is proven.
B Hardware and Software Environment
The experiments are performed on two Linux servers (CPU: Intel(R)
Xeon(R) CPU E5-2690 v4 @ 2.60GHz, Operation system: Ubuntu 16.04.6 LTS).
For GPU resources, two NVIDIA Tesla V cards are used for MLP and GNN and ResNet-50 on ImageNet experiments while two Titan V cards are used for other CNN models.
The python libraries we use to implement our experiments are
PyTorch 1.7.1 and PyG 1.6.3.
C Details of Reproduction C.1 Open Source Code
We publish our code in Github (i.e., https://github . com/lundu28/NSR ).
C.2 Details of Baseline Methods
In this subsection, we detailed the network architectures and the baseline methods used in our experiments.
MLP Multilayer Perceptron is a fundamental deep neural network architecture that is composed of an input layer, several hidden layers and an output layer.
Five MLP models with different numbers of hidden layers are used to test the performance of neuron steadiness regularization on MNIST for handwritten digit classification.
ReLU is used as an activation function in the experiments.
We denote
MLP-L as an L-layer MLP network.
For example, MLP-4 is a 4-layer MLP with 2 hidden layers.
CNN Convolutional neural networks are widely used in the computer vision (CV) field.
Two famous ones among them, ResNet-18 [11], VGG-19 [26], are chosen as the baseline methods in our paper for the image recognition task on CIFAR-10 data set.
ResNet-50 [11] is chosen for the ImageNet dataset [6].
ResNet-18 [11] is a 20-layer convolutional neural network with residual connections to avoid the problem of vanishing gradient.
ResNet-50 is similar to ResNet-18 except that it is a more expressive
50-layer convolutional neural network.
VGG-19 [42] is another classic deep convolutional neural network architecture.
We utilize a variant of VGG-19 adapted for CIFAR-10 [26] classification.
GNN Graph neural networks achieve fantastic results on graph structure data.
GCN [17] and
GraphSAGE are two classical methods in classifying nodes on real network data.
GCN [17] is inspired by CNN and introduces spectral graph convolutions as the layer-wise propagation on graphs.
GraphSAGE [10] is the first inductive GNN framework that samples neighbor nodes on graphs.
The detailed settings of different architectures of MLP and GNN are shown in Tab. 8.
Notice that we only show the architecture of hidden layers.
The dimensions of the input and output layer rely on the input feature size and the number of categories of different datasets respectively.
[Table]
[Table]
[Table]
Amazon
[Table]
C.3 Hyper-parameter Settings
We have only one unique hyperparameter, i.e., λ in our methods, and the detailed settings are given in the Tab. 9 and Tab. 10.
The search methods and search range are given in the main text.
All the other hyper-parameters are set the same as baselines.
C.4 Details of Datasets
In this section, we describe detailed information about the public available benchmark data sets used in our experiments.
MNIST [22] is a handwritten digits dataset containing 60000 training images and 10000 testing images.
It is used to examine the performances of MLP with different hidden layers.
CIFAR- [19] is used for ResNet-18 and VGG-19, which consists of 50,000 32 × 32 training color images and 10,000 testing images categorized as 10 classes.
ImageNet is a benchmark dataset used for
ResNet-50, which contains 14,197,122 annotated images with 1000 classes.
For GNN, we selected four real-world graph datasets: PubMed [48] is a paper citation network where nodes represent documents and edges represent citation links.
Both Amazon-Photo and Amazon-Computers [41] are subgraphs of Amazon co-purchase graph where nodes correspond to goods and edges connecting two nodes denote they are frequently bought together.
WikiCS [30] is derived from Wikipedia whose nodes correspond to computer science articles and edges are hyperlinks.
The license and source of these datasets are shown in Tab. 11 and Tab. 12, respectively.
All the datasets are public and widely used in many research works.
The collectors of the datasets are responsible for ensuring no privacy and consent issues.
[Table]
[Table]
[Table]
shchur/gnn-benchmark/raw/master/data/npz/
[Figure]
https://github.com/shchur/gnn-benchmark/raw/master/data/npz/
[Figure]
[Figure]
Vanilla
Vanilla
[Figure]
NSR
[Figure]
A v e r a g e v a r i a n ce
Vanilla
Vanilla+NSR
[Figure]
D Dynamics of GNN and CNN during training procedure
In this section, we study the training procedure of ResNet-18 (CNN) and GraphSAGE-2 (GNN) of the vanilla version and model with NSR.
The cross entropy loss and the expectation of neuron intra-class response variance are demonstrated in Fig 4 and Fig 5.
The results of GraphSAGE-2 and
ResNet-18 show a similar tendency with MLP-4 discussed in Section 2.2.
Both figures illustrate that the model with NSR shows smaller intra-class response variance which leads to high test accuracy shown in Sec 4.2.1.
The overall increasing tendency of intra-class variance on GraphSAGE-2 is similar to the tendency of MLP as expected.
However, there exist a few outliers.
A potential reason is that the sampling process of GraphSAGE will introduce noise in the generated features which leads to perturbed responses of individual neurons during the early training phase.
E Time and space complexity analysis
In this section, we elaborate on details of the time and space consumption of NSR.
The additional computation time overhead of adding NSR to the model is small.
Tab.
13 shows the comparison of time consumption for one training epoch between the vanilla models and models with NSR respectively on MLP, ResNet-18 (CNN) and GraphSAGE-2 (GNN).
We could see that the additional overhead is small especially for deeper networks like ResNet-18.
[Table]
The additional space consumption is also small.
The space complexity for NSR can be denoted as
O ( N × J × M ) , where N is the number of neurons in a layer, J is the number of categories and
M is the length of memory queue.
According to Fig. 6, the performance of MLP model is robust with the length of memory queue and a small memory queue length can still lead to a competing performance.
So the length of memory queue can be set as a small constant.
All experiments set the length of memory queue as 5.
Therefore, the computation space complexity can be directly denoted as O ( N × J ) .
We show that this space complexity is more negligible than that of vanilla model, including all model weights and intermediate outputs of every layer.
Using MLP as an example, the space complexity of weights of a layer can be represented as O ( N in × N ) , where N in represents the input feature dimension.
The space complexity of output for a layer can be denoted as O ( N × B ) , where B denotes batch size.
Since the number of categories J is usually smaller than the number
[Figure]
of neurons N in and batch size B , the space complexity of NSR
is smaller than that of weights and output for a layer.
The space used for NSR is usually much smaller than the space used for storing weights and layer output.