T HE INTRIGUING ROLE OF MODULE CRITICALITY IN THE GENERALIZATION OF DEEP NETWORKS
Anonymous authors
Paper under double-blind review
A BSTRACT
We study the phenomenon that some modules of deep neural networks (DNNs) are more critical than others.
Meaning that rewinding their parameter values back to initialization, while keeping other modules ﬁxed at the trained parameters, results in a large drop in the network’s performance.
Our analysis reveals interesting properties of the loss landscape which leads us to propose a complexity measure, called module criticality , based on the shape of the valleys that connects the initial and ﬁnal values of the module parameters.
We formulate how generalization relates to the module criticality, and show that this measure is able to explain the superior generalization performance of some architectures over others, whereas earlier measures fail to do so.
I NTRODUCTION
Neural networks have had tremendous practical impact in various domains such as revolutionizing many tasks in computer vision, speech and natural language processing.
However, many aspects of their design and analysis have remained mysterious to this date.
One of the most important questions is “what makes an architecture work better than others given a speciﬁc task?”
Extensive research in this area has led to many potential explanations on why some types of architectures have better performance; however, we lack a uniﬁed view that provides a complete and satisfactory answer.
In order to attain a uniﬁed view on superiority of one architecture over another in terms of generalization performance, we need to come up with a measure that effectively captures this.
Analyzing the generalization behavior of neural networks has been an active area of research since Baum & Haussler (1989).
Many generalization bounds and complexity measures have been proposed so far.
Bartlett (1998) emphasized on the norm of the weights in predicting the generalization error.
Since then various analysis have been proposed.
These results are either based on covering number and Rademacher complexity (Neyshabur et al., 2015; Bartlett et al., 2017;
Neyshabur et al., 2019; Long & Sedghi, 2019; Wei & Ma, 2019) or they use approaches similar to PAC-Bayes (McAllester, 1999; Dziugaite & Roy, 2017; Neyshabur et al., 2017; 2018; Arora et al., 2018; Nagarajan & Kolter, 2019a; Zhou et al., 2019).
Recently authors have emphasized on the role of distance to initialization rather than norm of the weights in generalization (Dziugaite &
Roy, 2017; Nagarajan & Kolter, 2019b; Neyshabur et al., 2019).
Earlier results have exponential dependency with depth and focus on fully connected networks.
More recently, Long & Sedghi (2019) provided generalization bounds for convolutional neural networks (CNNs) and fully connected networks used in practice and their bounds are size-free and have linear dependency on depth.
Despite the success of earlier works in capturing the dependency of generalization performance of a model with respect to different parameters, they fail to provide a ranking in terms of generalization performance for candidate architectures given a speciﬁc task that aligns well with the ground truth.
Moreover, majority of these bounds are proposed for fully connected modules and it is not straightforward to evaluate them for different architectures such as ResNets.
Every DNN architecture is a computation graph where each node is a module 1 .
We are interested in understanding the role of the fundamental component of a DNN, i.e., module, in generalization and capture how they interact with each other.
To do so, we delve deeper into the phenomenon of “module criticality” which was reported by Zhang et al. (2019a).
They observed that modules of the network present different robustness characteristics to parameter perturbation.
Speciﬁcally, if you rewind one module back to its initialization value while keeping the other modules ﬁxed at their trained value, the impact of this perturbation on network performance varies and depends on which module was rewound.
Some modules are “critical” meaning that rewinding their value to initialization harms the network performance, while for the others the impact of this perturbation on performance is negligible.
They show that various conventional architectures exhibit this phenomenon.
Let us now informally deﬁne what we mean by the measure “module criticality” (see Figure 1).
For each module, we move on a line from its ﬁnal trained value to its initialization value (convex combination 2 path) while keeping all other modules the same.
Then we measure the size of drop in the performance.
Let θ αi = (
[Equation]
−
[Equation]
) θ 0 i
+ αθ
Fi , α ∈ [0 , 1] be the convex combination between initial weights θ 0 and the ﬁnal weights θ F , where α i is the minimum value between 0 and 1 when performance (train error) of the network drops by at most a threshold value (cid:15) .
If α i is small we can move a long way back to initialization without hurting performance and the “module criticality” of this module would be low.
Further, we also wish to incorporate the robustness to noise (that is, the valley width) for the module along this path.
If the module is robust to noise along this path (valley is wide) then the module criticality would again be low (see Deﬁnition 3.1 for a formal deﬁnition).
In this paper, we seek to study this phenomenon in depth and shed some light on it by showing that conventional complexity measures cannot capture criticality, see Section 2.
Next we theoretically formulate this phenomenon and analyze its role in generalization.
Through this path, we provide a new generalization measure that captures the dissimilarity of different modules and point out how it inﬂuences the generalization performance of the corresponding DNN.
Intuitively, the closer we can get to initialization for each module, the better the generalization performance.
We analyze the relation between generalization and module criticality through PAC-Bayesian analysis.
We show that it is not the number of critical modules that matters, but rather the overall network criticality measure as derived in Section 3.
If network criticality measure is smaller for an architecture, it has better generalization performance.
In Section 4, we demonstrate through various experiments that our proposed measure is able to distinguish between different network architectures in their generalization performance, and the ranking it predicts matches their empirical performance, whereas earlier approaches fail to capture this.
Notation:
We use upper case letters for matrices.
The operator norm and Frobenius norm of M are denoted by (cid:107) M (cid:107) 2 , (cid:107) M (cid:107)
Fr respectively.
For n ∈ N , we use [ n ] to denote the set { 1 , . . . , n } .
Let
L S ( f ) be the loss of function f on the training set S with m samples.
We are mainly interested in the classiﬁcation task where L S ( f ) =
1 m (cid:80) ( x,y ) ∈ S 1 [ f ( x )[ y ] ≤ max j (cid:54) =
y f ( x )[ j ]] .
For any γ > 0 , we also deﬁne margin loss L S,γ ( f ) = 1 m (cid:80) ( x,y ) ∈ S 1 [ f ( x )[ y ] ≤ γ + max j (cid:54) =
y f ( x )[ j ]] .
Let L D ( f ) be the loss of function f on population data distribution D deﬁned similar to L S ( f ) .
We will denote the function parameterized by Θ by f Θ .
T OWARDS UNDERSTANDING MODULE CRITICALITY 2. S ETTING
A DNN architecture is a directed acyclic computation graph which may or may not be sequential.
In order to have a unifying deﬁnition between different architectures, we use the notion of “ module ”.
A module is a node in the computation graph that has incoming edges from other modules and outgoing edges to other nodes and performs a linear transformation on its inputs.
For a sequential model such as a VGG, a module is equivalent to a layer.
On the other hand, in a ResNet some modules are parallel to each other.
For example, a downsample module and the concatenation of two convolutional modules in a ResNet18-v1 architecture.
Note that, similar to conventional deﬁnitions the non-linearity is not part of the module.
Let Θ =
( θ 1 , . .
. , θ d ) correspond to all parameters of a DNN with d modules, where θ i refers to the weight matrix (or operator matrix in case of convolution) at module i and θ 0 i , θ Fi refer to the value of weight matrix at initialization and the end of training, respectively.
For sequential architectures, d is equal to the depth of the network but that is not necessarily true for a general architectures such as a ResNet.
2. R OBUSTNESS TO REWINDING
Consider the following perturbation to a trained network at some training epoch as in (Zhang et al., 2019a).
For each module in the network, rewind its value back to initialization while keeping the
[Figure]
values of all other modules the same.
Next, measure the change in performance of the model before and after this manipulation.
We repeat a similar analysis that differs from that by Zhang et al. (2019a) in one detail.
Zhang et al.
(2019a) rewind the whole resnet block at once, whereas we rewind each module, i.e. each convolutional module separately.
This rewind analysis is shown in Figure 2 for
ResNet18.
Each column represents a layer in ResNet18-v1 that has been rewound to the initialization value and each row represents a particular training epoch.
Similar to earlier analysis, we observe that for many modules of the network, this manipulation does not inﬂuence the network performance drastically, while for some others the impact is more pronounced.
For example, in Figure 2 we look at effect of rewinding on train error.
‘The ‘Stage2.block1.conv2” module is critical, whereas most other modules, once rewound, do not affect the performance.
In Figure 6 in Appendix D we depict the effect of rewinding on different performance criteria (train loss, train error and test error) and note that they exhibit a similar trend.
A stable phenomena: The plots in Figure 2 capture a network trained with SGD with weights initialized with the standard Kaiming initialization (He et al., 2015).
To ensure that the observed phenomenon is not an artifact of the training method and the initialization scheme, we repeated the experiments with different initialization and optimization methods.
We noted a similar pattern.
For example, Figures 7a, 7b in the Appendix illustrate the pattern when we changed the initialization to Fixup (Zhang et al., 2019b), and when we replace SGD with Adam (Kingma & Ba, 2014) respectively.
2. W HAT MEASURES FAIL TO DISTINGUISH CRITICAL LAYERS
Spectrum of weight matrices: To understand this phenomenon, we examine how weight matrices evolve during training to see if this can differentiate between critical and non-critical layers.
We explore the change in the spectrum of different weight matrices on rewinding and note that the spectrum for a critical and non-critical module look alike.
This is shown in Figures 3.
We calculate the spectrum of convolutional layers using the algorithm by Sedghi et al.
Distance to initialization: Next, we analyze the operator norm of difference from initialization for each module.
Figure 8 in the appendix depicts this and reveals no difference between critical and non-critical modules.
A similar plot was explored by Zhang et al.
(2019a), where they ﬁnd that
Frobenius norm and inﬁnity norm also fail to capture criticality.
Change in the activation patterns
We investigate the change in the activation patterns of a network when we rewind a module.
To do this we study the similarity between two networks: 1.
The original trained network and 2.
The network with a rewound module.
To measure similarity we use CKA (Kornblith et al., 2019).
We note that for a non-critical module, the original and rewound networks are similar and in case of a critical module, the similarity between the activation patterns between the two networks degrades gradually rather than abruptly.
See Figure 9 in Appendix D.
G ENERALIZATION B OUNDS BASED ON M ODULE C RITICALITY
Our goal is to understand criticality and how it affects the generalization performance of a DNN.
Inspired by the rewind to initialization experiments of Zhang et al. (2019a), we take one step further and consider changing the value of each module, to the convex combination of its initial and ﬁnal value, i.e., for each module i , we replace θ i with θ αi =
(1 − α )
θ
i + αθ
Fi , α ∈ [0 , 1] , and keep all other layers the same.
Then we look at the effect of this perturbation on the performance of the network.
Figure 4 depicts how train error, test error and train loss change as we increase the value of α in θ αi when i is a critical module (green curve), a non-critical module (orange curve) and all modules (blue curve).
We ﬁnd that along this convex combination path all these performance measures drop monotonically, as we move from the ﬁnal weights to the initial weights.
The above experiment shows the effect of moving along a convex combination between module’s initial and trained value.
To capture the relation between criticality and generalization, we are interested in also accounting for the width of the valley as we move from ﬁnal value to the initial value.
In particular, we are interested in analyzing what happens if we are moving inside a ball of some radius σ i around each point in this path.
PAC-Bayesian analysis, looks for a ball around ﬁnal value of parameters such that the loss does not change if we move in this ball.
Bringing this idea together with the one mentioned above, we are interested in moving from ﬁnal value to initialization value in a valley of some radius, and ﬁnd out how far back we can move on this path.
Intuitively, being able to move closer to initialization values indicate that the effective function class is smaller and hence the network should generalize better.
For example, in the extreme case where none of the weights change from their initialization value the function class would be a single function (the initial function) and the generalization error would be very low ( ∼ 0 % ) as both the train and test error would be very high (but equal).
Deﬁnition 3.1 (Module and Network Criticality) .
Given an (cid:15) > 0 and network f Θ , we deﬁne the module criticality for module i as follows:
[Equation]
We also deﬁne the network criticality as the sum over the modules of the module criticality:
[Equation]
Here, L S denotes the empirical zero-one loss over the training set, f θ αi , Θ F − i is the DNN’s function value where weight matrix corresponding to i th module is replaced by θ αi and all other modules are ﬁxed at their values in the end of training, Θ F − i .
θ αi =
(1 − α )
θ
i + αθ Fi ,
where θ 0 i is the value of the weight matrix at initialization and θ
Fi is the trained value.
Intuitively, network criticality measure is sum of module criticalities.
This is also theoretically derived using the analysis below.
3. A PAC-B AYESIAN G ENERALIZATION B OUND
We attempt to understand the relationship between module criticality, and generalization by deriving a generalization bound using the PAC-Bayesian framework (McAllester, 1999).
Given a prior distribution over the parameters that is picked in advanced before observing a training set, and a posterior distribution over the parameters that could depend on the training set and a learning algorithm, PAC-Bayesian framework bounds the generalization error in terms of the Kullback–Leibler (KL) divergence (Kullback & Leibler, 1951) between the posterior and the prior distribution.
We use PAC-Bayesian bounds as they hold for any architecture.
The intuition from Figure 4 suggests moving the parameters of each module as close as possible to initialization before harming the performance.
For such α i , we can then deﬁne the posterior Q i for module i to be a Gaussian distribution centered at θ α i with covariance matrix σ 2 i , i.e., as if we have additive noise u i ∼ N (0 , σ i I ) .
We use Θ α to refer to the case where all θ i are replaced with θ α
i i and matrix U includes all u i .
Then the following theorem holds.
Theorem 3.2.
For any data distribution D , number of samples m ∈ N , for any 0 < δ , for any 0 < σ i ≤ 1 and any 0 ≤ α ≤ 1 , with probability 1 − δ for the choice of the training set S m ∼ D the following generalization bound holds:
[Equation]
where k i is the number of parameters in module i .
For example, for a convolution module with kernel size q i × q i and number of output channels c i , k i =
q 2 i c i − 1 c i .
The exact bound including the constants and the proof of the above theorem is given in Appendix A.
Theorem 3.2 already gives us some insight into generalization of the original network.
However, it is not exactly a generalization bound on the original network but rather the perturbed network.
We conjecture that for almost any realistic distribution D , any random Θ 0 , any Θ F achieved by known gradient based optimization algorithms, any 0 ≤ α ≤ 1 and any σ ≥ 0 , the test error does not improve by taking a convex combination of parameters and their initial values followed by Gaussian perturbation.
Therefore, we have L D ( f Θ F ) ≤ E U
[ L D ( f Θ α + U )] .
The following corollary restates
Theorem 3.2 by using this assumption and doing some search over α and σ parameters in the bound.
Corollary 3.3.
For any data distribution D , number of samples m ∈ N .
For any (cid:15) > 0 , for any 0 < δ , if L D ( f Θ F ) ≤ E U [ L D ( f Θ α + U )] for u i ∼ N (0 , σ i I ) , then with probability 1 − δ for the choice of the training set S m ∼ D , the following generalization bound holds
[Equation]
where µ (cid:48) (cid:15) ( f Θ ) is calculated as follows:
[Equation]
Note that the above bound uses a slightly different notion of network criticality compare to Deﬁnition 3.1 since the bound requires ﬁnding α and σ values simultaneously for all modules as opposed to the one deﬁned in Deﬁnition 3.1 which allows us to decouple them.
Deterministic Generalization Bound for Convolutional Networks Although PAC-Bayesian bounds are data-dependent hence numerically superior, they provide less insight about the underlying reason that causes generalization.
For example, the ﬂatness of the solution after adding Gaussian perturbation can be computed numerically.
But computing its value does not reveal what properties in the network enforce the loss surface around a point to be ﬂat.
Deterministic norm-based generalization bounds on the other hand are numerically much looser yet provide better insights into dependency of generalization to different parameters.
In Appendix B, we build on the results of the
Theorem 3.2 to present a norm-based deterministic bound using module criticality.
In this section, we intuitively justiﬁed network criticality measure and proved the form of dependency of DNN generalization to network criticality measure in Corollary 3.3.
In Section 4, we empirically show that network criticality measure is able to correctly rank generalization performance of different architectures while earlier measures fail to do so.
E XPERIMENTS
We perform several experiments to compare our network criticality measure to earlier complexity measures in the literature.
All our experiments are performed on CIFAR10 and CIFAR100 datasets.
For all experiments, implementation and architecture details are presented in Appendix C.
Table 1 summaries the quantities that are calculated in this Section.
The quantity SoSP was proposed by Long & Sedghi (2019).
For the last two measures, we calculate σ i and α i as per Deﬁnition 3.1.
In our experiments, each module is a single convolutional or linear layer.
This represents a natural choice where each module is a linear transformation (with respect to the parameters).
This choice also leads to the lowest number of modules such that each module is a linear transformation.
[Table]
First as a sanity check we use our complexity measure (lower is better) to compare between a
ResNet18 trained on true labels and compare it with a ResNet18 trained on data where 20% of the labels are randomly corrupted.
As seen in Figure 5a, our measure is able to correctly capture that the network trained with true label generalizes better than the one trained on corrupted labels (4.62% error vs. 35% ).
Next, in Table 2 we compare the generalization performance of several conventional DNN architectures trained on the CIFAR10 dataset.
There is a particular ranking of the networks based on their generalization error and we want to capture this ranking by the complexity measure.
Therefore, we compare the rankings proposed by network criticality measure and complexity measures from the literature with the empirical rankings obtained in the experiment.
To do this we calculate the
Kendall’s τ correlation coefﬁcient (Kendall, 1938) which is deﬁned as follows:
Kendall’s
[Equation]
# of pairs where the rankings agree − # of pairs where the rankings disagree
[Equation]
This coefﬁcient lies between − 1 and 1 , where 1 denotes a high correlation between the two set of rankings.
Table 2 shows that the Kendall’s τ coefﬁcient between our network criticality measure and the generalization error is higher than all other complexity measures that we compared to.
We ﬁnd that our measure correctly ranks the generalization performance of the networks –
ResNet34, ResNet18, ResNet101, VGG16 and FCN (I).
It fails to correctly identify the correct rank of ResNet50, DenseNet121, VGG11 and FCN (II).
We also repeat the above experiment for the same networks trained on the CIFAR100 dataset (see
Table 3).
We note that network criticality measure correctly predicts the ranking of generalization
[Figure]
performance for ResNet101, ResNet34, ResNet18, ResNet50, VGG16 and the 3-layer fully connected networks (FCN).
We ﬁnd that the generalization error of ResNet101 is the lowest which is correctly captured only by our complexity measure and not by any other measures.
Further, the
Kendall’s τ correlation coefﬁcient between the ranking based of the generalization error and our network criticality measure is 0.55 which is again higher than this coefﬁcient for any other complexity measure.
Our measure only fails to capture the relative ranking of VGG11 and DenseNet relative to the other other DNN architectures and the relative ranking between FCN (I) and FCN (II).
[Table] [Table]
C ONCLUSION
In this paper we studied the module criticality phenomenon and proposed a complexity measure based on module criticality that is able to correctly predict the superior performance of some DNN architectures over others, for a speciﬁc task.
We believe module criticality can be used as a road-map for designing new task-speciﬁc architectures.
Proposing new regularizers that improve generalization performance by bounding criticality or spreading it among various modules of the network is an exciting direction for future work.
Our measure could also be potentially used in architecture search where we could calculate this score over the training set to select architectures that generalize well on the unseen test set.
A PPENDIX A P ROOF OF THE T HEREORM 3.
We start by stating the PAC-Bayes theorem which bounds the generalization error of any posterior distribution Q on parameters Θ that can be reached using the training set given a prior distribution
P on parameters that should be chosen in advance and before observing the training set.
Throughout this section given two scalar p, q ∈ [0 , 1] let KL ( p || q ) denote the KL divergence between two bernoulli distributions with success probabilities p and q respectively.
Theorem A.1 (McAllester (1999)) .
For any data distribution D , number of samples m ∈ N , training set S m ∼ D , and prior distribution P on parameters Θ , posterior distribution Q , for any 0 < δ , with probability 1 − δ over the draw of training data we have that
[Equation]
where KL is the the Kullback–Leibler (KL) divergence (Kullback & Leibler, 1951).
Following Dziugaite & Roy (2017), we use the inequality KL − 1 ( q | c ) ≤ q + (cid:112) c/ 2 to achieve a simple bound on the test error:
[Equation]
where KL − 1 ( q | c )
= sup { p ∈ [0 , 1] : KL ( q || p ) ≤ c } .
The intuition from Figure 4 suggests that moving the parameters of each module as close as possible to initialization before harming the performance.
For such α i , we can then deﬁne the posterior Q i for module i to be a Gaussian distribution centered at θ αi with covariance matrix σ 2 i , i.e., as if we have additive noise u i ∼ N (0 , σ i I ) .
We use Θ α to refer to the case where all θ i are replaced with θ α
i i and matrix U includes all u i .
Then the training loss term can be decomposed as
[Equation]
where the second term on the right hand side of the inequality captures the ﬂatness of the point Θ α by adding Gaussian noise and measuring the change in the loss.
Therefore, searching over the posterior corresponds to ﬁnding a ﬂat solution in the valley that connects the initial and ﬁnal points.
Next, we use this intuition to prove a generalization bounds based on module criticality.
First, we express the value of the (cid:101) O (1) term in Theorem 3.2 , which is equal to (cid:15) as below
[Equation]
Now we proceed with the proof.
The KL-divergence between two k -dimensional Gaussian distributions is given by the formula:
[Equation]
The above equation can be further simpliﬁed for Gaussian distributions with a diagonal covariance matrix.
Let the prior P be a Gaussian distribution such that for each module i , the distribution is
N ( θ 0 i , σ 2 P,i I ) and let the posterior Q be a Gaussian distribution such that for each module i , the distribution is N ((1 − α )
θ
i + αθ Fi ) , σ 2 Q,i I ) .
We can then write the KL-divergence KL ( Q || P ) as
[Equation]
Since prior should be decided before observing the training set, we are not allowed to optimize for σ P,i directly.
However, one can optimize for σ P,i over a pre-deﬁned set of values and use a union bound argument to get the generalization bound for the best σ P,i in that set.
We use a covering approach suggested by Langford & Caruana (2002).
For b, (cid:15) > 0 , if one chooses the variance of prior to be exp( − (cid:15)j + b ) for j ∈ N such that for each j the bound holds with probability 1 − 6 π 2 j 2 , then all bounds hold with probability 1 − (cid:80) j ∈ N
π 2 j 2 = 1 − δ .
We can apply the same idea to every module such that the bound holds with probability 1 − δ (cid:81) di =1
π 2 j 2 i .
If we choose σ 2 Q,i ≤ 1 then we have σ P,i ≤ exp (cid:16) 4 mk i + 1 (cid:17) .
Otherwise, the bound holds since the right hand side is greater than one.
Given (from Equation 3) (cid:15) ≥ 0 , if we choose σ 2 P,i to have the form exp (cid:16) 4 m − j i k
i + 1 (cid:17) , for some integer j
i
[Equation]
we can always ﬁnd choose j i such that
[Equation]
Therefore
[Equation]
the KL-divergence can be bounded as
[Equation]
Note that in order to achieve the inequality in Equation
[Equation]
j i should be chosen as j i =
[Equation]
Given that each such bound should hold with probability 1 − δ (cid:81) di =1 6 π 2 j 2 i , the log term in the bound can be written as
[Equation]
Putting everything together proves the Theorem statement.
B A D ETERMINISTIC G ENERALIZATION B OUND FOR C ONVOLUTIONAL N ETWORKS
We start by stating a generalization bound given by Neyshabur et al.
(2018) with a slight improvement in the constants.
Lemma B.1 (Neyshabur et al.
Let f Θ : X → R C be any predictor function with parameters Θ and P be a prior distribution on parameters Θ .
Then for any γ, m, δ > 0 , with probability 1 − δ over the training set S of size m , for any parameter Θ and any perturbation distribution Q over parameters such that P U ∼ Q (cid:2) max x ∈X | f Θ+ U ( x ) − f Θ ( x ) | ≤ γ 4 (cid:3) ≥ 12 , we have
[Equation]
The lemma above gives a data-independent deterministic bound which depends on the maximum change of the output function over the domain after a perturbation.
We combine the Lemma B. with the Theorem 3.2 and prove a bound on the perturbation which leads to the following theorem.
Theorem B.2.
Let norm of the input x an N × N image whose norm is bounded by B , f Θ : X → R C be the predictor function with parameters Θ which is a DNN of depth d made of convolutional blocks.
Then for any margin γ , sample size m , δ > 0 , with probability 1 − δ over the training set S , any parameter Θ and any α i > 0 such that max x ∈X | f Θ ( x ) − f Θ α ( x ) | ≤ γ 8 , we have
[Equation]
where k i is the number of parameters in module i .
For example, for a convolution module with kernel size q i × q i and number of output channels c i , k i =
q 2 i c i − 1 c i .
Proof.
First, we express the value of the (cid:101) O (1) term in Theorem B.2 , which is equal to (cid:15) 2 as below (cid:15)
[Equation]
We note that for any Θ , Θ (cid:48) , if max x ∈X (cid:107) f Θ ( X ) − f Θ (cid:107) ∞ ≤ γ/ 2 then L ( f Θ ) ≤ L γ ( f (cid:48) Θ ) .
The reason is that the output for each class can change by at most γ/ 2 and therefore the label can only change for the data points that are within γ of the margin.
We start using the assumptions on the perturbation bound.
Combining the results from Theorem B. and Theorem 3.2, we can get the following bound.
[Equation]
m
[Equation]
where (cid:15) 2 is given above in Equation 7.
Therefore, it sufﬁces to ﬁnd the value of σ i under which the assumption on norm of perturbation in function space holds and then simplify the following upper bound given the desired value of σ i .
In order to ﬁnd the desired value of σ i we use the following two lemmas.
First we adopt the perturbation lemma by Neyshabur et al.
(2018) to bound the change in the output a network based on the magnitude of the perturbation:
Lemma B.3 (Neyshabur et al.
Let norm of input x be bounded by B .
For any B > 0 , let f Θ : X → R C be a neural network with ReLU activations and depth d .
Then for any Θ , x ∈ X , and any perturbation
U s.t.
(cid:107) u
i (cid:107) 2 ≤ (cid:107) θ i (cid:107) 2 , the change in the output of the network can be bounded as follows
[Equation]
We next use the following lemma by Pitas et al.
(2017) that bounds the magnitude of the Gaussian perturbation u i for each convolutional module based on the standard deviation of the perturbation.
Lemma B.4 (Pitas et al.
Let u i be a Gaussian perturbation for each module i of a convolutional model.
Let N be the image size, q i , c
i be the kernel size and the number of output channels at module i respectively.
We have that
[Equation]
The lemma above suggests that by taking union bounds over all modules, we can ensure that with probability 1 / 2 we have that for any module i , the following upper bound on the spectral norm of the perturbation holds.
[Equation]
Combining this with perturbation bound in Equation 9, we have that
[Equation]
where the last inequality can be achieved with
[Equation]
Therefore, this value for σ i ensures the assumption on norm of perturbation in function space in
Theorem B.2 holds and hence completes the proof
[Equation]
Moreover
[Equation]
we show how we get the value of (cid:15) 2 , by showing the simpliﬁcation from inserting the value for σ i from Equation Equation 10 as follows.
[Equation]
32 edBα i (cid:13)(cid:13) θ
Fi − θ
i (cid:13)(cid:13)
Fr (cid:81) di
=1 (cid:107) θ αi (cid:107) 2 (cid:112) log(4 dN 2 )]
c i γ(cid:33) .
Then, (cid:15) 2 can also be simpliﬁed as follows.
[Equation]
 
[Equation]

[Equation]
 
[Equation]
C D ETAILS ON EXPERIMENTAL SETUP
For all our experiments, we use the CIFAR10 and CIFAR100 datasets.
To train our networks we used Stochastic Gradient Descent (SGD) with momentum 0 .
9 to minimize multi-class cross-entropy loss.
On CIFAR10 each model is trained until the cross-entropy loss on the training dataset falls below 0.15.
While on CIFAR100 each model is trained until the cross-entropy loss on the training dataset falls below 0.25.
The ResNets, DenseNets and VGGs were trained using a stage-wise constant learning rate scheduling with a starting learning rate of 0.1 and with a decrease by a multiplicative factor of 0.2 every 60 epochs.
FCN was trained with an initial learning rate of 0.1 with a decrease by a multiplicative factor of 0.2 every 200 epochs.
Batch size of 128 was used for all models and weight decay with factor 5e-4 was used to train all networks.
We mainly study three types of neural network architectures:
• Fully Connected Networks (FCNs): The FCNs consist of 2 fully connected layers.
FCN (I) contains 5000 and 1000 hidden units respectively while FCN (II) contains 10000 andhidden units respectively.
Each of these hidden layers is followed by a batch normalization layer and a ReLU activation.
The ﬁnal output layer (that follows the ReLU activation in the second layer) has an output dimension of 10 or 100 (number of classes).
• VGGs: Architectures by Simonyan & Zisserman (2015) that consists of multiple convolutional layers, followed by multiple fully connected layers and a ﬁnal classiﬁer layer (with output dimension 10 or 100).
We study the VGG with 11 and 16 layers.
• DenseNets:
Architectures by Huang et al.
(2017) that consists of multiple convolutional layers, followed by a ﬁnal classiﬁer layer (with output dimension 10 or 100).
We study the DenseNet with 121 layers.
• ResNets: Architectures used are ResNets V1 (He et al., 2016).
All convolutional layers (except downsample convolutional layers) have kernel size 3 × 3 with stride 1 .
Downsample convolutions have stride 2.
All the ResNets have ﬁve stages (0-4) where each stage has multiple residual/downsample blocks.
These stages are followed by a maxpool layer and a ﬁnal linear layer.
Here are further details about the ResNets used in the paper:
– ResNet18: ResNet18 architechtures studied in the paper have 1 convolutional layer in Stage 0 (64 ouput channels), Stage 1 has 2 residual blocks (64 output channels), Stage 2 has one downsample block and one residual block (128 output channels), Stagehas one downsample block and one residual block (256 output channels) and Stageagain has one downsample block and a residual block (512 output channels).
– ResNet34: ResNet34 architectures in this paper have 5 stages.
Stage 0 has 1 convolutional layer with 64 output channels followed by a ReLU activation.
Stage 1 hasresidual blocks (64 output channels), Stage 2 has 1 downsample block and 3 residual blocks (128 output channels), Stage 3 has 1 downsample block and 5 residual blocks (256 output channels) and, Stage 4 has 1 downsample block and 2 residual blocks (512 output channels).
– ResNet50: ResNet50 architectures in this paper again have 5 stages.
Stage 0 has 1 convolutional layer with 64 output channels followed by a ReLU activation.
Stage 1 has 1 downsample block and 2 residual blocks (256 output channels), Stage 2 has 1 downsample block and 3 residual blocks (512 output channels), Stage 3 has 1 downsample block and 5 residual blocks (1024 output channels) and, Stage 4 has 1 downsample block and 2 residual blocks (2048 output channels).
– ResNet101: ResNet101 architectures in this paper again have 5 stages.
Stage 0 hasconvolutional layer with 64 output channels followed by a ReLU activation.
Stagehas 1 downsample block and 2 residual blocks (256 output channels), Stage 2 hasdownsample block and 3 residual blocks (512 output channels), Stage 3 has 1 downsample block and 22 residual blocks ( 1024 output channels) and, Stage 4 has 1 downsample block and 2 residual blocks (2048 output channels).
The ResNets, DenseNets and VGGs in the paper are trained without batch normalization.
During training, images are padded with 4 pixels of zeros on all sides, then randomly ﬂipped (horizontally) and cropped.
Global mean and standard deviation are computed on all training images and applied to normalize the inputs.
While training a ResNet18 on the CIFAR10 dataset with 20% of the labels randomly corrupted, we do not augment the training set with images that are randomly ﬂipped and cropped.
We also do not use weight decay during training these networks.
D F IGURES
[Figure]
[Figure] [Figure]