Neuron with Steady Response Leads to Better Generalization
Abstract
Regularization can mitigate the generalization gap between training and inferenceby introducing inductive bias.
Existing works have already proposed various inductive biases from diverse perspectives.
However, none of them explores inductivebias from the perspective of class-dependent response distribution of individualneurons.
In this paper, we conduct a substantial analysis of the characteristics ofsuch distribution.
Based on the analysis results, we articulate the Neuron Steadiness
Hypothesis: the neuron with similar responses to instances of the same class leadsto better generalization.
Accordingly, we propose a new regularization methodcalled Neuron Steadiness Regularization (NSR) to reduce neuron intra-class response variance.
Based on Complexity Measure, we theoretically guarantee theeffectiveness of NSR for improving generalization.
We conduct extensive experiments on Multilayer Perceptron, Convolutional Neural Network, and Graph Neural
Network with popular benchmark datasets of diverse domains, which show thatour Neuron Steadiness Regularization consistently outperforms the vanilla versionof models with signiﬁcant gain and low additional computational overhead.
Introduction
Deep Neural Network (DNN) achieves state-of-the-art results in a wide range of areas and has variousapplications across industries, including self driving cars [37], virtual assistants [38], intelligenthealthcare [32], personalized recommendation [34], etc.
DNN’s success usually relies on a plentyamount of training data.
However, DNN’s generalization is often hampered in many domains wheretraining data is insufﬁcient because data annotation is labor-intensive and expensive.
Regularization is a popular method which helps to improve generalization via introducing inductivebias.
Regularization is one of the key elements of machine learning, particularly of deep learning[8].
Speciﬁcally, inductive bias represents assumptions about the model properties other thanthe consistency of outputs with targets.
There have been tremendous efforts in identifying suchdesired properties, which results in a series of widely used regularization methods.
For example, L
Regularization [36, 21] penalizes large norms of model weights, which puts constraints on “parameterscale".
L1 regularization improves “sparseness" by rewarding zero weight or neuron response.
Jacobian regularization [43, 13] minimizes the norm of the input-output Jacobian matrix to improve“smoothness" of the learned mapping function.
Orthogonal regularization [4, 1] enlarges “weightdiversity" to reduce the feature redundancy.
Batch Normalization [15] promotes “training dynamics stability" by reducing the internal covariate shift.
Although the existing works have already proposed various inductive biases from diverse perspectives, including aforementioned “parameter scale", “sparseness", “smoothness", “weight diversity",“training dynamics stability", to the best of our knowledge, there is no work to explore inductive bias from the perspective regarding the characteristics of neuron response distribution on each class
From another point of view, the existing works leverage the information related to weights (parameter scale regularization), weight correlations (orthogonal regularization), derivatives of mappingfunction (smoothness regularization), collective neurons responses (sparseness regularization, Batch
Normalization), but none of them considers the intra-class response distribution of individual neurons.
[Figure] [Figure]
In this paper, we study the characteristics of the intra-class response distribution of each individualneuron to identify the new regularization method.
In more detail, for each individual neuron, weanalyze the variance of its response to samples of the same class, which is called neuron intra-classresponse variance .
We ﬁnd that such intra-class response variance has an obvious correlation withclassiﬁcation correctness.
As shown in Figure 1, we ﬁnd the correctly classiﬁed samples usuallyhave smaller intra-class response variance compared to the misclassiﬁed samples.
Besides, it can beobserved that the vanilla model with cross entropy as the optimization target usually could not controlintra-class response variance well, as shown in Figure 2, which leaves a potential improvement spacefor the regularization.
The details of experiments and observations are explained in Section 2.
Based on these observations, we articulate a Neuron Steadiness Hypothesis : neuron with similarresponses to instances of the same class, i.e., smaller neuron intra-class response variance can lead tobetter generalization.
Accordingly, we propose the regularization method called Neuron Steadiness
Regularization (NSR) to improve generalization by penalizing large neuron intra-class responsevariance.
Based on Complexity Measure theory in Section 3.3, we conduct a theoretical analysis on theeffectiveness of NSR for improving generalization.
In addition, our regularization method showssigniﬁcant improvements on various network architectures, including Graph Neural Network (GNN),
Convolution Neural Network (CNN), and Multilayer Perceptron (MLP).
To sum up, our contributions are as follows:
• We articulate a Neuron Steadiness Hypothesis and demonstrate its validity. It provides a new regularization perspective based on the neuron-level class-dependent response distribution.
• We propose a new regularization method, Neuron Steadiness Regularization, to improvegeneralization ability.
The method is computationally efﬁcient and general enough to beapplied to various architectures and tasks.
Theoretical analyses guarantee its effectiveness.• Extensive experiments are conducted on multiple types of datasets like images, citationgraphs and product graphs, with various network architectures including GNN, CNN andMLP.
Signiﬁcant improvements evidently verify the effectiveness of the Neuron SteadinessRegularization.
Observations
In this section, we verify the Neuron Steadiness Hypothesis by experiments with the followingidentiﬁed observations.
2.1 Correlation between neuron intra-class response variance and classiﬁcation correctness
The neuron intra-class response variance is derived from neuron response distribution which could beobtained by recording neuron responses when we feed input samples to the model.
In this paper, forthe neuron with ReLU as its activation function, we do not take its zero-response into account forcalculation, because zero-response corresponds to the inactivated state where the neuron does notrespond at all.
In other words, for neurons with ReLU as the activation function, we only record itsnon-zero responses to represent its response distribution.
With the obtained response distribution of any given neuron, it is relatively straightforward to calculateour discussed statistics, i.e., intra-class response variance, which is the mean squared deviation ofthe neuron response from the mean of the intra-class response.
Then, for each neuron, we calculateintra-class response variance corresponding to the correctly classiﬁed samples and misclassiﬁedsamples, respectively.
Finally, we aggregate such two respective variances of all neurons in thepenultimate layer separately and present them in Figure 1.
Figure 1 shows that for different networks, the average intra-class response variance of correctlyclassiﬁed samples is smaller than that of misclassiﬁed ones on arbitrary class.
It indicates the strongcorrelation between classiﬁcation correctness and neuron intra-class response variance
.2.2 Dynamics of neuron intra-class response variance during training procedure
We investigate the tendency of neuron intra-class response variance along with the training procedure.
For comparison, we perform the analysis on the vanilla model and the model with our proposedneuron steadiness regularization.
We calculate the intra-class response variance of each neuron onthe entire training set after each training epoch.
Then, the intra-class response variance of all neuronsare averaged and denoted as average variance in Figure 2 (c).
We also show the testing accuracy andthe training cross entropy loss in Figure 2 (a) and (b), respectively.
Other architectures demonstratesimilar tendencies and can be found in Appendix D.
From Figure 2, we could see that the cross entropy objective keeps being optimized during thetraining procedure which leads to increasing classiﬁcation accuracy for both models.
However, forthe vanilla version, the average neuron intra-class response variance is growing larger because themodel training does not impose constraints on neuron intra-class response variance.
For the modeltrained with our proposed regularization, the neuron intra-class response variance is controlled anddecreases
after a few epochs.
More importantly, the learned model with well-controlled intra-classresponse variance has higher testing accuracy than the vanilla version although its correspondingcross entropy loss is even larger.
In addition, we could also see that regulating neuron intra-classresponse variance may also help the optimization procedure to achieve higher accuracy in earlierepochs.
To conclude all above observations, it is reasonable to design the regularization based on the neuronsteadiness hypothesis, i.e., reducing neuron intra-class response variance, for better generalization.
Method
In this section, we ﬁrst describe the proposed Neuron Steadiness Regularization .
Then we furtherintroduce several techniques for computational efﬁciency.
3.1 Deﬁnition of Neuron Steadiness Regularization
Neuron Steadiness Regularization (NSR) for a speciﬁc neuron is deﬁned as the summation ofintra-class response variances of different classes.
The NSR for the n -th neuron can be formulated as:
[Equation]
where X n,j is a random variable denoting the n -th neuron’s response for a sample belonging to thej -th class.
J is the number of classes.
↵ j = z j P
i z i is the prior probability of the j -th class where z jis the sample amount of j -th class.
Notice that ↵ j is not a hyper-parameter, and it only presents theimportance of different classes.
With the NSR term deﬁned for each individual neuron, the overallregularization is derived by applying NSR term on all neurons in the network as follows:
[Equation]
where L S represents the NSR term for the entire network, N is the number of neurons in the network. n is the hyper-parameter to control the regularization intensity.
In this paper, for practical simplicity, is set as the same value for all neurons.
Adding the overall regularization term to the main trainingtarget, i.e., the cross entropy loss, the ﬁnal regularized loss function can be written as:
[Equation]
where L C represents cross entropy loss.
3.2 Practical Implementation
3.2.1 Mini-batch Training
In order to use mini-batch training, we adapt our NSR method to do forward and backward propagationbased on mini-batches of samples.
To be speciﬁc, we ﬁrst transform Eq.
(1) as:
[Equation]
The ﬁrst term has the same form as typical loss functions, i.e., an expectation of the function ofdata samples, which can be easily generalized to a mini-batch training setting via estimating theexpectation with a batch of samples.
Although the second term can be estimated in the same way, thesquare operation magniﬁes the estimation error, especially when the batch size is not large enough.
To alleviate the problem while not introduce much computing overhead, we propose a memory queuebased estimation method that allows us to leverage more history samples for estimation withoutadditional sampling and forward/backward computation.
For each class, we record the number ofsamples and the summation values of the neurons’ response within each batch.
To further reduce thestorage overhead, we only maintain these values of the latest M batches by two M -length queues.
More speciﬁcally, an element c m,j in the ﬁrst memory queue is the count number of j -th classinstances in the m -th batch, represented as:
[Equation]
where Y m is the set of labels in the m -th batch, y i is the label of i -th sample, and  is a characteristicfunction, i.e.,  ( condition )
= 1 if condition is satisﬁed, otherwise  ( condition ) = 0 .
An elements ( n ) m,j in the second memory queue is the summation value of n -th neuron’s response for the samplesbelonging to the j -th class within the m -th batch, represented as:
[Equation]
) where X ( n ) m is the set of the n -th neuron’s response within the m -th batch and x ( n ) i is the n -th neuron’sresponse of the i -th sample.
When a new batch is fed, the estimation of expectation E [ X n,j ] can beupdated by the following steps:
[Equation]
where ˆ E [ X n,j ] is the estimation of expectation E [ X n,j ] , C j = P m c m,j , S ( n ) j =
P m s ( n ) m,j , andc ⇤ ,j , s ( n ) ⇤ ,j , as new elements appended to the queues, represent the count number and the summation forthe new batch.
Based on this dynamic update method, the additional memory overhead is negligible,and we give a space complexity analysis in Appendix E.
3.2.2 Layer Selection criterion for Applying NSR
We ﬁnd that there is the correlation or redundancy among the steadiness constraints of differentlayers, meaning that applying NSR on different layers of network has an overlapping effect onneuron steadiness control.
In more details, from Figure 3, we can see that every layer’s variance ratiodecreases even only one speciﬁc layer is applied with NSR.
Such steadiness correlation or redundancyamong layers is not surprising because different layers are served as input/output of each other.
Due to the aforementioned redundancy, considering the trade-off between performance gain andcomputational overhead, we select only one layer to apply NSR.
As our NSR is used to reduce neuronintra-class response variance, naturally, the intuitive criterion is to apply NSR to the layer with thelargest aggregated neuron intra-class response variance.
Detailed experiment results in RQ4 of theexperiment section show such criterion works well.
In this paper, if not otherwise speciﬁed, we apply
NSR to only one particular layer determined by this layer selection criterion.
3.3 Theoretical Analysis
In this section, we theoretically analyze the difference in model generalization ability with or withoutour proposed regularization method NSR based on Complexity Measure [35].
Complexity Measureis one of the mainstream methods to measure the generalization ability of a deep learning model.
Alower complexity measure means a better generalization ability .
Formally, a Complexity Measureis a measure function M : {H , S} ! R + where H is a class of models and S is a training set.
According to the deﬁnition, if a complexity measure of a given model tending to be 0, the modelshould have the best generalization ability.
Note that, several different complexity measures havebeen proposed.
We select Consistency of Representations [33] as the concrete Complexity Measure used in ourtheoretical analysis.
This measure is designed based on Davies-Bouldin Index [5], and is the
Winning Solution of the NeurIPS 2020 Competition on Predicting Generalization in Deep Learning.
Mathematically, for a given dataset and a model, we deﬁne the following statistics:
[Equation]
where i and j are two different classes, and n i is the number of samples belonging to the class i .
O ( ⌧ ) i is the output representation of the ⌧ -th sample belonging to class i for the ﬁnal layer, and µ O i isthe cluster centroid of the representations of class i .
S i is a measure of scatter within representations
[Figure]
of class i , and M i,j
is a measure of separation between representations of classes i and j .
Then, thecomplexity measure based on the Davies-Bouldin Index is deﬁned as:
[Equation]
Based on the deﬁnition of the complexity measure C , we have the following lemma:
Lemma 3.1.
For a multi-class classiﬁcation problem, when (1) a deep learning model utilizing the
Cross Entropy loss with an NSR regularization term on any of its intermediate layer is optimized viagradient descent and (2) the capacity of the model is sufﬁciently large, the Consistency of Representations Complexity Measure on this model C will tend to be 0.
Under the same condition, when thedeep learning model is only optimized by the Cross Entropy loss without an NSR regularization term,there will be inﬁnite local minima where the complexity measure C will be a positive number.
The proof is shown in Appendix A. Intuitively, this lemma shows that in an ideal condition (i.e.,the model is sufﬁciently large), if the deﬁnition of the Consistency of Representations Complexity
Measure is utilized, a deep learning model with NSR regularization term will be guaranteed toconverge to local minima with the best generalization ability.
In contrast, the model without NSRwill not have the same guarantee.
Experiment
In this section, we conduct experiments over a variety of datasets to examine the performance ofour neuron steadiness regularization on three extensively used neural network architectures.
Wedesign a series of experiments to answer following research questions and publish our code in link.
RQ1 : How does NSR perform on various datasets and different neural architectures?
RQ2 : Does
NSR outperform other classical regularization methods?
RQ3 : What is the effect of combining NSRwith other popular methods like Batch Normalization or Dropout?
RQ4 : What if we apply NSR tomultiple layers instead of the particular one layer?
4.1 Experiment Setup
Network Architectures and Datasets .
Three architectures utilized in experiments are Multilayer
Perceptron (MLP), Convolutional Neural Network (CNN) and Graph Neural Network (GNN).
Forvanilla models used as our baselines in different architectures, we adopt ResNet-18 [11], VGG-19[26] and ResNet-50 for CNN, GCN [17] and GraphSAGE [10] for GNN.
We run ﬁve MLP modelswhich are denoted as MLP-L, L indicates the number of layers including input layer.
We elaborateon the details of these vanilla models in Appendix C.2.
As for benchmark datasets used in ourexperiments, MLP and CNN are applied to image recognition task on MNIST [22], CIFAR-10 [19]and ImageNet [6] datasets, respectively.
GNN is applied to node classiﬁcation on four real-worldgraph datasets: WikiCS [30], PubMed [48], Amazon-Photo and Amazon-Computers [41].
Noticethat ImageNet is a large benchmark dataset with 1000 classes.
Details are shown in Appendix C.4.
[Table] [Table]
Experiment Settings .
We follow the typical implementation settings to conduct ourexperiments.
For ResNet-18 and VGG19 on CIFAR-10, we follow the detailedsetting of [49, 9].
For ResNet-50 on ImageNet, we follow the ofﬁcial implementation provided by torchversion library 1 .
For
GraphSAGE and GCN, we follow the implementation setting of [28].
For MNISTdataset, we divide 60000 training images into the training set with 50000 samples and the validationset with the remaining 10000 samples to select hyper-parameter.
For each of four graph datasets, itis randomly split into training, validation [Table] and testing sets with ratio 6:2:2.
Note that, SGD [40] isused to optimize MLP, and Adam [16] for other models except for ResNet-18 that is optimized by
Momentum [40] according to the implementation setting of [49].
We use the typical setting of batchsize as 100 for all experiments.
To ensure the model convergence, training epochs are set as 100 forboth MLP and GNN, and 200 for ResNet and 500 for VGG.
In all experiments except for RQ4, we apply NSR to only one particular layer with using the same for each neuron.
For this only one hyper-parameter  , like most of other regularization methods,we apply random search strategy to ﬁnd its proper value ranging from 1e-2 to 10.
The error rateis the metric and each result is averaged over 5 runs with different random seeds.
The hardwareenvironments are detailed in Appendix B.
4.2 Experiment Results
RQ1: Performance of NSR : Here, we discuss the performance of NSR over different models andpresent the results on Tab. 1 - 3 “Gain” is the percentage of relative reduction in the error rate.
Tab.1 demonstrates that NSR can improve the performance of MLP with different layers: the relativeerror rate is reduced by 9.39% at least and 43.79% at most.
Besides, as the number of network layersincreases, the gain of NSR shows a roughly upward trend.
The four-layer MLP achieves the lowestclassiﬁcation error rate among all vanilla version baselines, and the accuracy becomes worse as thenetworks grow deeper.
It indicates that deep MLP encounters a severe overﬁtting problem.
Thesuccess of our regularization in addressing such a problem reveals the importance of stabilizing theresponse of each individual neuron to instances from the same class.
For CNN models, Tab. 2 demonstrates that NSR can reduce the relative error rate for CIFAR-10 by11.97% on VGG-19 and 11.37% on Resnet-18, and reduce the relative error rate (top-5) for ImageNetby 10.74% on ResNet-50.
It is worth mentioning that VGG-19, ResNet-18 and ResNet-50 havealready adopted Batch Normalization, Dropout regularization and Weight Decay in vanilla models,the accuracy gain of our method reveals that NSR can have extra beneﬁts to ulteriorly enhancegeneralization ability when combining with Batch Normalization, Dropout and Weight Decay.
Wewill show more evidence about this in RQ3.
The number of layers of two GNN models varies from 2 to 4.
It is a typical setting following empiricalexperiences as nodes will capture similar information from neighbors and result in over smoothnesswhen GNNs grow deeper.
Tab.
3 shows that GraphSAGE and GCN applied with NSR outperform thevanilla model with different layer depths on all datasets.
Speciﬁcally, GraphSAGE and GCN achievean average improvement of 8.6% and 5.8%, respectively, and 17.0% improvement at most.
RQ2: Comparison with Other Regularization : The vanilla ResNet18 here does not use anyregularization methods for fairness. We ﬁrst compare our NSR method with several classical
[Table] [Table]
[Table] [Table]
[Table]
regularization methods shown in Tab. 4. Notice that, both our NSR and the other regularizationmethods for comparison have only one hyper-parameter to tune, and we utilize the same hyperparameter searching strategy to select the best hyper-parameter values for all of them. As shown in
Tab. 4, our NSR performs best among these regularization methods on all three models with differentarchitectures, i.e., MLP-4 (MLP), ResNet-18 (CNN), GraphSAGE (GNN).
The results indicate thatour NSR has remarkable improvement and is general for various architectures.
RQ3: Combination with Other Regularization : We further investigate the effect of combiningour NSR with other popular methods like Batch Normalization and Dropout.
We conduct theexperiments on MLP-4 and the result are organized in Tab. 5.
From Tab. 5 we can ﬁnd that both
Batch Normalization and Dropout can reduce error rate compared with vanilla baseline, and addingour NSR on the top of them can further promote the performance signiﬁcantly.
It indicates that NSRcan provide complementary regularization beneﬁts with Batch Normalization and Dropout.
RQ4: Layer selection for applying NSR : We have introduced how to select one particular layer toapply NSR in section 3.2.2.
To evaluate such layer selection criterion, we compare the performancestrained by applying NSR to the multiple layers and to the only one selected layer, respectively.
Taking
MLP-4 on MNIST and GraphSAGE-2 on Amazon-Computers as two examples, we apply NSR todifferent layer(s), and their corresponding error rates are listed in Tab.
6 ⇠ Tab. 7, where model l means NSR is applied to the l th layer of the model.
Notice that, except the ﬁrst layer of MLP-4, theinput layer, for the second, third, and last layer of MLP-4, their neuron intra-class response varianceare 409, 510, and 1660, respectively.
For GraphSAGE-2, its neuron intra-class response variance are4.15 and 2.68, for the ﬁrst and last layer, respectively.
The variance of MLP-4 and GraphSAGE-2 arequite different because of the data characteristic difference between MNIST and Amazon-Computers.
The results in Tab. 6 - 7 show that, no matter which layer(s) is applied with NSR, it could alwaysimprove the accuracy compared with the vanilla baseline for both MLP-4 and GraphSAGE-2.
Inaddition, according to our layer selection criterion, applying NSR to the layer with the biggestvariance, i.e., the last layer for MLP-4 and the ﬁrst layer for GraphSAGE-2, could achieve the mostsigniﬁcant gain compared with applying NSR to other individual layers.
Also, it could achievesimilar accuracy compared with the best one obtained by applying NSR to multiple layers.
This isthe empirical evidence to demonstrate the rationality of our layer selection criterion.
Related Work
Regularization improves generalization ability by introducing the inductive bias based on priorknowledge.
According to the type of prior knowledge, existing works can be roughly categorizedinto Domain Speciﬁc Regularization and Model Generic Regularization.
Domain Speciﬁc Regularization shows success in various domains including Face Recognition[46, 24, 2, 25], Knowledge Graph Completion [50, 20, 31], Graph Neural Network [3, 14, 39, 47, 27].
In Face Recognition, [2] points out the existence of external factors, such as different situations ofenvironmental illumination, head poses, facial expressions, which brings the challenge that faceimages from the same person may have even larger differences than face images from differentpersons.
To address speciﬁc challenges in face recognition, [24] imposes the careful-designedregularization method which enforces the representations of face instances from the same person tobe similar.
In Graph Neural Network, the popular model usually suffers from the over-smoothingproblem due to the six degrees of separation [18].
[3] statistically analyzes the high correlationbetween smoothness and the mean average distance (MAD) among node representation.
Then aregularization called MADGap is proposed which punishes over smoothness by minimizing MAD.
Model Generic Regularization can be categorized into network-wise Regularization, layer-wise
Regularization, and neuron-wise Regularization, according to the granularity of the studied properties.
Network-wise Regularization encodes the desired property of the entire network, like the sparsenessof the network, the smoothness of the mapping function between input and output.
[-L
 Regularization [35, 20]-]L2 Regularization[36, 21] encourages small sum squared magnitude of model weights.
[13] introduces an efﬁcientframework to minimize the norm of input-out Jacobian matrix for noise robustness.
[7] encouragesmodel parameters to be uniformly low at some local of loss function.
Layer regularization [45,36] becomes popular due to the success achieved by Batch Normalization [15].
Neuron-wise Regularization is the method with the most ﬁne-grained granularity.
Dropout [12, 44] is the typicalexample belonging to this category.
It randomly removes neurons with a certain probability to avoidstrong co-adaption between neurons.
Activation Regularization [29] adds the [-L
 Regularization on-]L2 Regularizationon masked activations of neurons and Temporal Activation Regularization restricts the differencebetween RNN outputs at adjacent timesteps.
To minimize the variance of sample variance and obtaina few distinct modes, Variance Constancy Loss [23] is applied on all neurons before the activationas regularization.
Our proposed Neuron Steadiness Regularization belongs to the neuron-wiseregularization that leverages information of individual neuron response distribution.
Conclusion and Future Work
We explore the inductive bias from the new perspective of class-dependent response distributionof individual neurons.
Based on experimental observations, we articulate the Neuron Steadiness
Hypothesis and propose the Neuron Steadiness Regularization that penalizes large intra-class neuronresponse variance.
Based on Complexity Measure, we provide theoretical analysis on its effectivenessfor improving generalization.
Additionally, we conduct extensive evaluations on diverse datasetswith various network architectures to demonstrate its power.
Especially, we show its effectivenesson the classiﬁcation task with large model like ResNet-50, and large dataset with many classes like ImageNet with 1000 classes.
We systematically consider the border impact, No risk is found.