E MPIRICAL S TUDY OF THE D ECISION R EGION AND R OBUSTNESS IN D EEP N EURAL N ETWORKS
Anonymous authors
Paper under double-blind review
A BSTRACT
In general, Deep Neural Networks (DNNs) are evaluated by the generalization performance measured on unseen data excluded from the training phase.
Along with the development of DNNs, the generalization performance converges to the state-of-the-art and it becomes difﬁcult to evaluate DNNs solely based on this metric.
The robustness against adversarial attack has been used as an additional metric to evaluate DNNs by measuring their vulnerability.
However, few studies have been performed to analyze the adversarial robustness in terms of the geometry in DNNs.
In this work, we perform an empirical study to analyze the internal properties of DNNs that affect model robustness under adversarial attacks.
In particular, we propose the novel concept of the Populated Region Set (PRS) , where trained samples are populated more frequently, to represent the internal properties of DNNs in a practical setting.
From systematic experiments with the proposed concept, we provide empirical evidence to validate that a low PRS ratio has a strong relationship with the adversarial robustness of DNNs.
I NTRODUCTION
With the steep improvement of the performance of Deep Neural Networks (DNNs), their applications are expanding to the real world, such as autonomous driving and healthcare (LeCun et al., 2015; Miotto et al., 2018; Huang & Chen, 2020).
For real world application, it may be necessary to choose the best model among the candidates.
Traditionally, the generalization performance which measures the objective score on the test dataset excluded in the training phase, is used to evaluate the models (Bishop, 2006).
However, it is non-trivial to evaluate DNNs based on this single metric.
For example, if two networks with the same structure have the similar test accuracy, it is ambiguous which is better.
Robustness against adversarial attacks, measure of the vulnerability, can be an alternative to evaluate
DNNs (Szegedy et al., 2013; Goodfellow et al., 2014; Gu & Rigazio, 2014; Huang et al., 2015;
Jakubovitz & Giryes, 2018; Yuan et al., 2019; Zhong et al., 2021).
Adversarial attacks aim to induce model misprediction by perturbing the input with small magnitude.
Most previous works were focused on the way to ﬁnd adversarial samples by utilizing the model properties such as gradients with respect to the loss function.
Given that the adversarial attack seeks to ﬁnd the perturbation path on the model prediction surface over the input space, robustness can be expressed in terms of the geometry of the model.
However, few studies have been performed to interpret the robustness with the concept of the geometric properties of DNNs.
From a geometric viewpoint, the internal properties of DNNs are represented by the boundaries and the regions (Baughman & Liu, 2014).
It is shown that the DNNs with piece-wise linear activation layers are composed of many linear regions, and the maximal number of these regions is mathematically related to the expressivity of DNNs (Mont´ufar et al., 2014; Xiong et al., 2020).
As these approaches only provide the upper bound for the expressivity with the same structured model, it does not explain how much information the model actually expresses.
In this work, we investigate the relationship between the internal properties of DNNs and the robustness.
In particular, our approach analyzes the internal characteristics from the perspective of the decision boundary (DB) and the decision region (DR), which are basic components of DNNs (Fawzi et al., 2017).
To avoid insensitivity of the maximal number of linear regions in the same structure assumption, we propose the novel concept of the Populated Region Set (PRS) , which is a set of DRs
[Figure]
containing at least one sample included in the training dataset.
Since the PRS can be considered as the feasible complexity of the model, we hypothesize that the size of PRS is related to the robustness of network.
To validate our hypothesis, we perform systematic experiments with various structures of DNNs and datasets.
Our observations are summarized as follows:
• The models with the same structure can have different size of PRS, although they have similar generalization performance.
In experiments, we observe that this difference leads to different robustness of the network.
• We empirically show that the size of the PRS is related to the robustness against the adversarial attack.
The model with a small size of the PRS tends to show higher robustness compared to that with a large size (in Section 4.1).
We further observe that when the model achieves a low PRS ratio, the linear classiﬁer that maps the penultimate features to the logits has high cosine similarity between parameters corresponding to each class (in Section 4.2).
• We verify that the size of intersection of the PRS from the training/test dataset is related to the robustness of model.
The model with a high PRS inclusion ratio of test samples has higher robustness than that with a low PRS inclusion ratio (in Section 5).
• We identify that the model with a small size of the PRS learns the sparse feature representation.
In quantiﬁcation, we observe the inversely correlated relationship between the size of the PRS and sparsity of feature representation (in Section 6).
R ELATED WORK
Adversarial robustness For the real-world application of DNNs, the adversarial attack, which reveals the vulnerability of DNNs (Goodfellow et al., 2014), is mainly used to validate the reliability of the trained network.
As an early stage for adversarial attacks, the fast gradient sign method (FGSM) (Goodfellow et al., 2014) based on the gradient with respect to the loss function and the multi-step iterative method (Kurakin et al., 2016) are proposed to create adversarial examples to change the model prediction with a small perturbation.
Recently, many studies on effective attacks in various settings (e.g., white-box or black-box) have been performed to understand the undesirable decision of the networks (Shaham et al., 2018;
Madry et al., 2018; Chen et al., 2020).
In terms of factors affecting robustness, Yao et al.
(2018) provide evidence to argue that training with a large batch size can degrade the robustness of the model against the adversarial attack from the perspective of the
Hessian spectrum.
In contrast, Kamath et al.
(2019) propose that the model with a constant ratio between the learning rate and batch size does not degrade the model robustness even with a large batch size as it converges to the ﬂatter minima.
Geometric Analysis inside Deep Neural Networks
With increasing interest in the expressive power of DNNs, there have been several attempts to analyze DNNs from a geometric perspective (Dauphin et al., 2014; Choromanska et al., 2015).
In these studies, the characteristics of the decision boundary or regions formulated by the DNNs are mainly discussed.
Mont´ufar et al.
(2014) show that the cascade of the linear layer and the nonlinear activation organizes the numerous piece-wise linear regions.
They show that the complexity of the decision boundary is related to the maximal number of these linear regions, which is determined by the depth and the width of the model.
Xiong et al.
(2020) extend the notion of the linear region to the convolutional layers and show the better geometric efﬁciency of the convolutional layers.
Fawzi et al.
(2018) reveal that classiﬁcation regions in DNNs are topologically connected and the decision boundary of natural images is ﬂat in most directions.
It has also been shown that the manifolds learned by DNNs and the distributions over them are highly related to the representation capability of a network (Lei et al., 2018).
While these studies highlight the beneﬁts of increasing expressivity of DNNs as the number of regions increases, interpreting the vulnerability of DNNs with the geometry is another important topic.
Yang et al.
(2020) show that a model with thick decision boundaries induces robustness.
Moosavi-Dezfooli et al.
(2019) show that a decision boundary with a small curvature acquires the high robustness of the model.
These approaches focus on the decision boundaries, while this paper suggests to focus on the decision regions, which are composed by the surrounding decision boundaries.
I NTERNAL P ROPERTY OF DNN S
This section describes the internal properties of DNNs from the perspective of decision boundaries (DBs) and regions (DRs).
The DBs of the DNN classiﬁer is mainly deﬁned as the borderline between
DRs for classiﬁcation, where the prediction probability of class i and the neighboring class j are the same (Fawzi et al., 2018).
To expand the notion of DBs and DRs to the internal feature-level, we redeﬁne the DBs in the classiﬁer that generalizes the existing deﬁnition of DBs.
We then propose the novel concept of the Populated Region (PR) that describes the speciﬁc DRs used from the network for training samples.
PR is used to analyze the relationship between the trained parameters and the characteristics of networks.
3. D ECISION B OUNDARY AND R EGION
Let the classiﬁer with L number of layers be F ( x ) =
f L ( σ ( f L − 1 σ ( · · · σ ( f 1 ( x ))))
=
f L :1 ( x ) , where x is the sample in the input space X ⊂ R D x and σ ( · ) denotes the non-linear activation function 1 .
For the l -th layer, f l ( · ) denotes the linear operation and f il :1 ( · ) denotes the value of the i -th element of the feature vector f l :1 ( x ) ∈ R D l .
We deﬁne the DB for the i -th neuron of the l -th layer.
Deﬁnition 1 (Decision Boundary (DB))
The i -th decision boundary at the l -th layer is deﬁned as
B il = { x | f il :1 ( x ) = 0 , ∀ x ∈ X} .
We note that the internal DB B il ( l < L ) divides the input space X based on the hidden representation of the l -th layer (i.e., existence of feature and the amount of feature activation).
There are a total of
D l boundaries and the conﬁguration of the DBs are arranged by the training.
As input samples in the same classiﬁcation region are considered to belong to the same class, the input samples placed on the same side of the internal DB B il share the similar feature representation.
In this sense, we deﬁne the internal DR, which is surrounded by internal DBs.
Deﬁnition 2 (Decision Region (DR)) Let V l ∈ {− 1 , +1 } D l be the indicator vector to choose positive or negative side of decision boundaries of the l -th layer.
Then the decision region DR V l , which shares the sign of feature representation, is deﬁned as
[Equation]
Figure 1 presents the internal properties for two networks trained on CIFAR-10 with similar test accuracy.
The right column in Figure 1 depicts the internal DBs and DRs in the network with a high/low PRS ratio (top and bottom).
We randomly select two test images (blue and green box) and generate adversarial images for blue box (orange and purple box) in each network, respectively.
We make a hyperplane with these images to visualize the DBs and DRs in the 2D space.
We identify that the conﬁguration of DBs and DRs appears to be different, although the two networks have the same structure and similar test accuracy.
3. P OPULATED R EGION S ET
It is well-studied that the number of DRs is related to the representation power of DNNs (Mont´ufar et al., 2014; Xiong et al., 2020).
In particular, the expressivity of DNNs with partial linear activation function is quantiﬁed by the maximal number of the linear regions and this number is related to the width and depth of the structure.
We believe that although the maximal number can be one measure of expressity, the trained DNNs with ﬁnite training data 2 cannot handle the entire regions to solve the task.
To only consider DRs that the network uses in the training process, we devise the train-related regions where training samples are populated more frequently.
We deﬁne the Populated Region Set (PRS), which is a set of DRs containing at least one sample included in the training dataset.
PRS will be used to analyze the relationship between the geometrical property and the robustness of DNNs in a practical aspect.
Deﬁnition 3 (Populated Region Set (PRS))
From the set of every DRs of the model f and given the dataset X , the Populated Region Set is deﬁned as
[Equation]
We can then deﬁne a Populated Region as a union of decision regions in PRS as
[Equation]
,f,l ) DR.
We note that the size of the PRS is bounded to the size of given dataset X .
When | PRS ( X , f, l ) | = | X | , each sample in training dataset is assigned to each distinct DR in the l -th layer.
To compare the
PRS of networks, we deﬁne the PRS ratio, | PRS ( f, X ,l ) | |X| , which measures the ratio between the size of the PRS and the given dataset.
[Figure]
Figure 2 presents a comparison between two equivalent neural networks (A and B) with six convolution blocks (CNN-6) trained on CIFAR-10 varying only the batch size (2048/128).
Figure 2 (a) presents the PRS ratio for the depth of layers in each model at the 300th epoch.
We observe that only the penultimate layer ( l = 8 ) shows a different PRS ratio.
Figure 2 (b) shows that the two networks have largely different PRS ratios with similar training/test accuracy.
From the above observation and the fact that the penultimate layers are widely used as feature extraction, we only consider the
PRS ratio on the penultimate layer in the remainder of the paper.
Experimental setups For the systematic experiments, we select three different structures of DNNs to analyze: (1) a convolutional neural network with six convolution blocks (CNN-6); (2) VGG-
(Simonyan & Zisserman, 2014); and (3) ResNet-18 (He et al., 2016).
We train 3 basic models with ﬁxed ﬁve random seeds and four batch sizes (64, 128, 512 and 2048) over three datasets: MNIST (LeCun & Cortes, 2010), F-MNIST (Xiao et al., 2017), and CIFAR-10 (Krizhevsky et al., 2009).
For the extensive analysis on the correlation between the PRS ratio and properties of network, we extract candidates from each basic model with the grid of epochs. Then we apply the test accuracy threshold to guarantee the sufﬁcient performance.
Finally, we obtain 947 models for analysis. The details for the network architecture and the selection procedure are described in Appendix A-C.
R OBUSTNESS U NDER A DVERSARIAL A TTACKS
In this section, we perform experiments to analyze the relationship between the PRS ratio and the robustness.
We evaluate the robustness of the network using the fast gradient sign method (FGSM)
(Goodfellow et al., 2014), basic iterative method (BIM) (Kurakin et al., 2016) and projected gradient descent (PGD) (Madry et al., 2018) method widely used for the adversarial attacks.
The untargeted adversarial attacks using training/test dataset are performed for the various perturbation ( (cid:15) ).
4. PRS AND R OBUSTNESS FOR M ODELS
First, we compare the two models (Network A and B in Figure 2) with similar test accuracy but different PRS ratio.
4 Figure 3 presents the results of robust accuracy (RA) under the FGSM, BIM (5-step), PGD-20 (20-step), and PGD-100 (100-step) on L ∞ . For each step, α = 2 / 255 is adopted.
We identify that Network B (low PRS ratio) is more robust than Network A (high PRS ratio) under all adversarial attacks.
[Figure]
obu s t
A cc u r ac y FGSM BIM [Figure] As the PGD attack with a 20-step size shows the similar RA compare to a 100-step size, we focus on an analysis under the PGD attack with a 20-step size in the rest of the paper.
We measure the PRS ratio and the RA in all models and datasets to verify the relationship between the PRS ratio and the robustness.
For the experiments, we take the magnitude of (cid:15) as follow: MNIST = 0.3, F-MNIST = 0.1, and CIFAR10 = 0.0313 on L ∞ norm.
Figure 4 presents the experimental results according to the model structure under the PGD attack.
To quantify the relation, we calculate the slope of the regression line and perform signiﬁcance test to validate the trend. We also provide the result of RA against the FGSM attack and AutoAttack (Croce & Hein, 2020) in Appendix G. From Figure 4, we identify that the PRS ratio has an inversely correlated relationship with the RA in most cases.
The complicated models (VGG-16 and ResNet-18) have lower PRS ratios compared to CNN-6, and show even lower PRS ratios on the simple datasets (i.e., MNIST and F-MNIST).
PRS and Robustness for Training We also analyze how the PRS ratio and the RA change during the training.
We measure the PRS ratio and the RA under the PGD attack for three different networks trained on CIFAR-10.
Figure 5 presents the test accuracy, PRS ratio and RA for each training epoch.
Although the test accuracy of each model converges after the 50th epoch, the PRS ratio continues to decrease after the 50th epoch and the RA continues to increase. [Figure] [Figure]
4. PRS AND F INAL L AYER
From the observations above, we empirically conﬁrm that the PRS ratio is related to the robustness against adversarial attacks.
In order to investigate the evidence that the low PRS ratio causes robustness for the gradient-based attack
[Figure]
we perform an additional analysis of failed attack samples.
In the gradient-based attack, as the magnitude of the gradient is a crucial component to success, we ﬁrst count the ratio of the zero gradient samples in the failed attack samples.
The Figure 6 (a) shows the ratio of success samples (light green bar), failure samples with non-zero gradient (blue bar) and zero gradient (red bar) in all samples.
We note that the failed attack samples with non-zero gradients maintain the index of the largest logit as the true class after attack.
To analyze the reason of failure, we examine the change of the logits under the adversarial attack.
This change is shown in Figure 6 (b).
To clarify the difference of the change of the logits between Network A and B, we select the examples of successful attack on Network A but failed attack on Network B.
In Network B, the logits move on almost parallel direction, which causes the predicted label to be maintained as the true class.
To explain the parallel change of the logit vector, we hypothesize that the DBs corresponding to each class node have similar conﬁguration in the input space.
However, it is intractable to measure the similarity between DBs in the entire network due to the highly non-linear structure and the high dimensional input space.
To simplify our hypothesis, we only measure the cosine similarity between the parameters which map the features on the penultimate layer to logits (i.e., ﬁnal layer).
Figurepresents that the similarity matrices for Networks A and B. When we compare the matrix between the two models at the 300th epoch, we identify that Network B (low PRS ratio) has greater cosine similarity between each parameter in the ﬁnal layer.
We note that the cosine similarity between each parameter in the ﬁnal layer can be considered as the degree of parallelism for the normal vectors in the linear classiﬁer.
We also conﬁrm that the decrease of the PRS ratio is aligned with the increase of the similarity of parameters in the Figure 7 (b), when we consider the graph in Figure 2.
To verify the relationship between PRS ratio and the cosine similarity between the parameters in the ﬁnal layer, we measure the PRS ratio and the cosine similarity between each parameter in all models.
Figure 8 shows the experiment results grouped according to the model structure.
We identify that
[Figure]
N e t A .
N e t B .
O r g .
N e t A . N e t B . Class index Class index
Logits (True class: 1)
Logits (True class: 0) Class index Logits (True class: 7) N e t A .
N e t B .
N e t A .
N e t B .
[Figure]
the PRS ratio has an inverse correlation for the cosine similarity between each parameter in the ﬁnal layer.
[Figure]
[Figure]
PRS AND T EST D ATASET
Regarding the model as a mapping function from the input space to the feature space, handling unseen data in a known feature domain is signiﬁcant with regard to the generalization performance of the model.
Hence, if the majority of samples from the test dataset are assigned to the training PR, the model can be considered to learn the informative and general concept of feature mapping.
For example, if the arbitrary test sample is mapped to the training PR, we expect that a similar decision will appear.
However, it is non-trivial to guess which type of decision will appear when the test sample is mapped to out of the training PR.
[Figure]
To investigate the differences between the test samples which are included/excluded in the training
PR, we evaluate the test accuracy under adversarial attack for each group.
For a controlled comparison, we divide both the inclusion and exclusion groups with 1k correctly predicted test samples.
Figure 9 shows the RA under the FGSM, BIM with a 5-step size, and the PGD with 20 and 100 step size on L ∞ .
Although the accuracy of each test group decreases as the epsilon becomes larger, we observe that the inclusion group is more robust against all types of attacks compared to the exclusion group.
We also provide the result of other networks and datasets in Appendix I.
Figure 10 presents the PRS ratio and the inclusion ratio of the test samples for the training PR.
We compute the inclusion ratio as the ratio of the test samples mapped to the training PR.
In Figure 10, we identify that the PRS ratio and the inclusion ratio exist in inversely correlated relationship.
As we previously verify that the included test samples show high robustness, we empirically conﬁrm that the low PRS ratio is related to the robustness under adversarial attacks.
[Figure]
PRS AND T RAINED F EATURES
This section explores the trained features of the models with the different PRS ratios.
First, we visualize the feature maps directly for each depth of layers.
Figure 11 shows illustrative examples of feature maps.
We identify that the model with the low PRS ratio learns more sparse features compared to the model with the high PRS ratio.
As the sparse features are considered as an independent and informative representation (Ranzato et al., 2007; Lee et al., 2007), if the PRS ratio can cause sparse feature representation in various cases, we conclude that the PRS ratio is related to the informative features.
To verify our hypothesis, we measure the trend between the PRS ratio and the average sparsity for each network.
The average sparsity of the model is calculated by taking the average of the ratio of zero-valued features over the training dataset for all layers.
Figure 12 shows the relationship between the PRS ratio and the average sparsity in all cases.
We identify that CNN-6 and VGG-16 show an inversely correlated relationship throughout the dataset, but it is difﬁcult to ﬁnd a clear relationship for ResNet-18.
We conjecture that (1) skip connection, or [Figure] N e t w o r k A N e t w o r k B
N e t w o r k A
N e t w o r k B
Depth of Layer ++ [Figure] (2) batch normalization can cause this phenomenon. To investigate the global feature representations further, we visualize the projected feature space using t-SNE (van der Maaten & Hinton, 2008) for the training dataset.
In Figure 6, we can identify that the samples in low PRS ratio cases are more densely located in each class, while the samples show widely spread pattern in each class in high
PRS ratio cases.
[Figure]
CNN-6 trained with various datasets.
The colored code indicates the class.
The samples in low PRS ratio cases are more densely located in each class compared to the samples in high PRS ratio.
C ONCLUSION
In this work, we perform an empirical study to analyze the internal properties of DNNs, which affect the robustness under adversarial attacks.
We propose the novel concept populated region set to derive the relationship between the internal properties of DNNs and robustness in a practical setting.
From systematic experiments for the proposed concept, we suggest that the PRS ratio is related to the robustness of DNNs and provide empirical evidence of this relationship: (1) The network with the a low PRS ratio shows high robustness against the gradient-based attack compared to the network with a high PRS ratio.
In particular, the model with a low PRS ratio has a higher degree of parallelism for the parameters in the ﬁnal layer, which can support robustness.
The network with a low PRS ratio includes more test samples in the training PR.
We empirically verify that this inclusion ratio is related to robustness from the observation that included test samples are more robust than excluded test samples.
The network with a low PRS ratio learns sparse feature representation to solve tasks.
Moreover its global feature representation is more intensive than the network with a high PRS ratio.