R EPRESENTATION Q UALITY E XPLAINS A DVERSARIAL A TTACKS
Anonymous authors
Paper under double-blind review
A BSTRACT
Neural networks have been shown vulnerable to adversarial samples.
Slightly perturbed input images are able to change the classiﬁcation of accurate models, showing that the representation learned is not as good as previously thought.
To aid the development of better neural networks, it would be important to evaluate to what extent are current neural networks’ representations capturing the existing features.
Here we propose a way to evaluate the representation quality of neural networks using a novel type of zero-shot test, entitled Raw Zero-Shot.
The main idea lies in the fact that some features are present on unknown classes and that unknown classes can be deﬁned as a combination of previous learned features.
To evaluate the soft-labels of unknown classes, two metrics are proposed.
One is based on clustering validation techniques (Davies-Bouldin Index) and the other is based on soft-label distance of a given correct soft-label.
Experiments show that such metrics are in accordance with the robustness to adversarial attacks and might serve as a guidance to build better models as well as be used in loss functions to create new types of neural networks.
Interestingly, the results suggests that dynamic routing networks such as CapsNet have better representation while current deeper DNNs are trading off representation quality for accuracy.
I NTRODUCTION
Adversarial samples are slightly perturbed inputs that can make neural networks misclassify.
They are carefully crafted by searching for variations in the input that, for example, could decrease the soft-labels of the correct class.
Since they were discovered some years ago (Szegedy, 2014), the number of adversarial samples have grown in both number and types.
Random noise was shown to be recognized with high conﬁdence by neural networks (Nguyen et al., 2015), universal perturbations, that can be added to almost any image to generate an adversarial sample, were shown to exist (Moosavi-Dezfooli et al., 2017), and the addition of crafted patches was shown to cause networks to misclassify (Brown et al., 2017).
Only one pixel is enough to make networks misclassify (Su et al., 2017).
Such attacks can also be easily transferred to real-world scenarios (Kurakin et al., 2016),(Athalye & Sutskever, 2018), which confers a big issue as well as a security risk for current deep neural networks’ applications.
Albeit the existence of many defences, there is not any known learning algorithm or procedure that can defend against adversarial attacks consistently.
Many works have tried to defend by hiding or modifying the gradients to make neural networks harder to attack.
However, a recent paper shows that most of these defences fall into the class of obfuscated gradients which have their shortcomings (e.g., they can be easily bypassed by transferable attacks) (Athalye et al., 2018).
Additionally, the use of an augmented dataset with adversarial samples (named adversarial training) is perhaps one of the most successful approaches to construct robust neural networks (Goodfellow et al., 2014),(Huang et al., 2015), (Madry et al., 2018).
However, it is still vulnerable to attacks and has a strong bias in the type of adversarial samples used in training (Tramèr et al., 2018).
This shows that a deeper understanding of the issues is needed to enable more consistent defences to be created.
Few works focused on understanding the reason behind such lack of robustness.
In (Goodfellow et al., 2014), it is argued that Deep Neural Networks’s (DNN) linearity are one of the main reasons.
Recent investigations reveal that attacks are changing where the algorithm is paying attention (Vargas & Su, 2019), other experiments show that deep learning neural networks learn false structures that are easier to learn rather than the ones expected (Thesing et al., 2019) and an accuracy and robustness trade-off for models were shown to exist (Tsipras et al., 2019).
In this paper, we propose a methodology of how to evaluate the representation of machine learning methods.
Based on these metrics, we reveal a link between deep representations’ quality and attack susceptibility.
Speciﬁcally, we propose a test called Raw Zero-Shot and two metrics to evaluate
DNN’s representations.
1. R ECENT A DVANCES IN A TTACKS AND D EFENSES
DNNs were shown vulnerable to many types of attacks.
For example, the output high conﬁdence results to noise images (Nguyen et al., 2015), universal perturbations in which a single perturbation can be added to almost any input to create an adversarial sample are possible (Moosavi-Dezfooli et al., 2017), the addition of image patches can also make them misclassify (Brown et al., 2017).
Moreover, the vulnerability can be exploited even with a single pixel, i.e., changing a single pixel is often enough to make a DNNs misclassify (Su et al., 2017).
Most of these attacks can be transformed into real-world attacks by merely printing the adversarial samples (Kurakin et al., 2016).
Moreover, crafted glasses (Sharif et al., 2016) or even general 3d adversarial objects (Athalye & Sutskever, 2018) can be used as attacks.
Although many defensive systems were proposed to tackle the current problems, there is still no consistent solution available.
Defensive distillation in which a smaller neural network squeezes the content learned by the original DNN was proposed (Papernot et al., 2016).
However, it was shown not to be robust enough (Carlini & Wagner, 2017).
Adversarial training was also proposed as a defence, in which adversarial samples are used to augment the training dataset (Goodfellow et al., 2014),(Huang et al., 2015), (Madry et al., 2018).
With adversarial training, DNNs increase slightly in robustness but not without a bias towards the adversarial samples used and while still being vulnerable to attacks in general (Tramèr et al., 2018).
There are many recent variations of defenses in which the objective is to hide the gradients (obfuscated gradients) (Ma et al., 2018), (Guo et al., 2018)
(Song et al., 2018). However, they can be bypassed by various types of attacks (such as attacks not using gradients, transfer of adversarial samples, etc.) (Athalye et al., 2018),(Uesato et al., 2018).
There are a couple of works which are trying to understand the reason behind such lack of robustness.
To citep some, in (Goodfellow et al., 2014), it is argued that the main reason may lie in DNNs’ lack of non-linearity.
Another work argues that the perturbations cause a change in the saliency of images which makes the model switch the attention to another part of it (Vargas & Su, 2019).
False structures that are easier to learn were also shown related to the problem (Thesing et al., 2019).
Moreover, in (Tsipras et al., 2019), the accuracy and robustness trade-off was shown to exist.
1. Z ERO -S HOT LEARNING
Zero-Shot learning is a method used to estimate unknown classes which do not appear in the training data.
The motivation of Zero-Shot learning is to transfer knowledge from training classes to unknown classes.
Existing methods approach the problem by estimating unknown classes from an attribute vector deﬁned manually.
Attribute vectors are annotated to both known and unknown classes, and for each class, whether an attribute, such as “colour” and “shape”, belongs to the class or not is represented by 1 or 0.
For example, in (Lampert et al., 2009) the authors proposed Direct Attribute Prediction (DAP) model, which learns each parameter for estimating the attributes from the target data.
It estimates an unknown class of the source data which is estimated from the target data by using these parameters.
Based on this research, other zero-shot learning methods have been proposed which uses an embedded representation generated using a natural language processing algorithm instead of a manually created attribute vector (Zhang & Saligrama, 2016; Fu et al., 2015; Norouzi et al., 2013; Akata et al., 2015; Bucher et al., 2016).
In (Zhang & Saligrama, 2015), a different approach to estimate unknown classes is proposed.
This method constructs the histogram of known classes distribution for an unknown class.
In this approach, it is assumed that the unknown classes are the same if these histograms generated in the target domain and the source domain are similar.
This perspective is similar to our approach because our method approach to represent an unknown class as the distribution of known classes.
However, our objective is not estimating the unknown class, and we do not use the source domain.
Our objective here is to analyze DNNs’ representation by using this distribution.
O N THE L INKS BETWEEN R OBUSTNESS E VALUATION AND R EPRESENTATION
In the canonical classiﬁcation setting, the goal of a classiﬁer is to achieve low expected loss: [Equation] Robustness against adversarial attacks is a slightly different setting.
To achieve a high robustness in this setting, a classiﬁer should have a lower adversarial loss in noise δ ∈ ∆ 1 :
[Equation] Considering Mean Squared Error (MSE), we have: [Equation] where h ( x ) =
E [ h D ( x )] is the expected behavior of the prediction when averaged over many datasets, f ( x ) is the ground-truth and ˆ y ( x ) = h D ( x ) is the output after learned on a given dataset D . For robustness to increase, adversarial training requires that datasets should have many noisy samples, i.e., x + δ ∈ D . However, the more noise is added to images the more D becomes close to all possible images R M ∗ N :
[Equation] However, f ( x ) is undeﬁned 2 for D ∈ R M ∗ N in which y / ∈ C for the set of known classes C . Even a small amount of noise may be enough to cause y / ∈ C and thus f ( x ) undeﬁned. Would it be possible to evaluate the robustness and/or the quality of a model without a well deﬁned y ?
To answer this question we take into account an ideal representation z and the representation learned by the model ˆ z : [Equation] Interestingly, although y is undeﬁned, z represents the features learned and is well deﬁned for any input. Moreover, by considering learned classes to be clustered in z space, unsupervised learning evaluation can be used to evaluate z even without a well deﬁned y .
We use here a famous clustering analysis index to evaluate clusters in z by their intracluster distance.
In z , it is also possible to evaluate the representation of known and unknown classes which should share some features. Moreover, we hypothesize here that unknown classes should evaluate z with less bias because a direct map from input to output is inexistent. Any projection of the input in any of the feature maps or the output layer could be used as z . To take the entire projection into account, we use here z as the ﬁnal projection of the input to the classes, i.e. z is the soft label array e .
R AW Z ERO -S HOT
In this paper, we propose to evaluate the representation learned by conducting experiments over the soft-labels of the image in unknown classes. This is based on the hypothesis that if a model is capable of learning useful features, an unknown class would also trigger some of these features inside the model. We call this type of test over unknown classes and without any other information, Raw Zero-Shot (Figure 1).
Feature(x)
Class(y) bear zebra bird train [1,0,0] [0,1,0] [0,0,1] test (soft-label)
[Table] [Table] [Table]
[Table] [Table]
[Table] [Table]
[Figure]
[Figure]
The Raw Zero-Shot is a supervised learning test in which only n − 1 of the n classes are shown to the classiﬁer during training.
The classiﬁer also has only n − 1 possible outputs.
During testing, only unknown classes are presented to the classiﬁer.
The soft-labels outputted for the given unknown class is recorded, and the process is repeated for all possible n classes, removing a different class each time.
To evaluate the representation quality, metrics computed over the soft-labels are used.
These metrics are based on a different hypothesis of what deﬁnes a feature or a class.
In the same way that there are different types of robustness, there are also different types of representation quality.
Therefore, metrics are somewhat complementary, each highlighting a different aspect of the whole.
The following subsections deﬁne two of them.
3. D AVIES -B OULDIN M ETRIC - C LUSTERING H YPOTHESIS
Soft labels of a classiﬁer compose a space in which a given image would be classiﬁed as a weighted vector concerning the previous classes learned.
Considering that a cluster in this space would constitute a class, we can use clustering validation techniques to evaluate the representation (Figure 2).
Here we choose for simplicity one of the most used metric in internal cluster validation, DaviesBouldin Index (DBI).
DBI is deﬁned as follows:
[Equation]
in which cn is the centroid of the cluster, e is one soft-label and n e is the number of samples.
3. A MALGAM M ETRIC - A MALGAM H YPOTHESIS
If DNNs can learn the features present in the classes, it would be reasonable to consider that the sof-labels also describe a given image as a combination of the previously learned classes.
This is also true when an image contains an unknown class.
Similar to a vector space in linear algebra, the soft-labels can be combined to describe unknown objects in this space.
This is analogous to how children describe previously unseen objects as a combination of previously seen objects.
Differently from the previous metric, here we are interested in the exact values of the soft-labels.
However, what would constitute the correct soft-labels for a given unknown class needs to be determined.
To calculate the correct soft-label of a given unknown class (amalgam proportion) automatically, we use here the assumption that accurate classiﬁers should output a good approximation of the amalgam proportion already.
Therefore, if a classiﬁer is trained in the n classes, the soft-labels of the remaining n − 1 classes is the amalgam proportion (Figure 2 illustrates the concept).
Consequently, the Amalgam Metric (AM) is deﬁned as:
[Equation]
in which, e’ is the normalized (such that they sum to one) soft-label from the classiﬁer trained over n classes and e is the soft-labels from the classiﬁer trained over n − 1 classes.
R AW Z ERO -S HOT E XPERIMENTS
Here, we conduct Raw Zero-Shot experiments to evaluate the representation of DNNs.
To obtain results over a wide range of architectures, we chose to evaluate CapsNet (a recently proposed completely different architecture based on dynamic routing and capsules) (Sabour et al., 2017), ResNet (a state-of-the-art architecture based on skip connections)(He et al., 2016), Network in Network (NIN) (an architecture which uses micro neural networks instead of linear ﬁlters) (Lin et al., 2013), All Convolutional Network (AllConv) (an architecture without max pooling and fully connected layers)(Springenberg et al., 2015) and LeNet (a simpler architecture which is also a historical mark) (LeCun et al., 1998).
All the experiments are run over the CIFAR dataset by using a training dataset with all the samples of one speciﬁc class removed.
This process is repeated for all classes, removing the samples of a different class each time.
To analyze the correlation between representation metrics and robustness against adversarial attacks, we conducted adversarial attacks on all the architectures tested using the most well known algorithms such as Carlini (Carlini & Wagner, 2017), Fast Gradient Method (FGM) (Goodfellow et al., 2014), Basic Iterative Method (BIM )(Kurakin et al., 2016), DeepFool (Moosavi-Dezfooli et al., 2016), Projected Gradient Descent Method (PGDM) (Madry et al., 2018)
(Table 1). For all tests, ε is ﬁxed to the corresponding value given in the table.
However, different methods have different meanings for ε : (a) for pixel attacks, ε is the maximum number of pixels to be changed, (b) for threshold attacks, ε is the maximum amount of change allowed per pixel, (c) for FGM, BIM and
PGDM, ε is the attack step size (input variation) and (d) for DeepFool, ε is the overshoot parameter.
Table 2 shows the ranking based on the attack accuracy and their required perturbation (Table 1).
In general, CapsNet is shown to be the most robust, AllConv and DenseNet follow with a proper [Table] [Table] placement while the remaining networks vary depending on the perspective used to analyze (i
[Table]
e., higher accuracy or lower L 2 ).
These robustness rankings will be used in the next sections to verify the relationship of robustness against adversarial samples and metrics to evaluate the representation quality.
4. E XPERIMENTS ON DBI M ETRIC
Table 3 shows the results with the DBI metric (the smallest the better) and the respective ranking of each neural network.
According to this metric, CapsNet possesses the best representation of all networks tested.
LeNet is considered the second-best neural network regarding representation, followed by AllConv, NIN and then deeper neural networks.
DBI metric matches exceptionally well with both accuracy and L 2 ranking based on robustness against adversarial samples.
Most differences in Hamming distance lies in the exact places in which both accuracy and L 2 rankings differ.
Notice that DBI metric does not use anything related to attacks and still arrives at similar rankings.
To further demonstrate the correlation between DBI and adversarial attacks, a Pearson correlation of the DBI for each network is shown in Table 4.
This table suggests that DBI and adversarial attacks have a statistical signiﬁcant correlation.
The fact that LeNet and other relatively more uncomplicated networks achieve a high representation quality which is at odds with accuracy may seem extremely unlikely. However, as discussed in (Tsipras et al., 2019), accurate models can trade-off robustness for accuracy.
DBI suggests that this trade-off happens because the representation quality has worsened.
Interestingly, LeNet and other [Table] [Table] simple networks are also easier to attack (low-rank accuracy) but need more perturbation to achieve the same accuracy (high rank L 2 ).
Therefore, LeNet and other simple networks might be easier to attack because the search space is less complicated (less obfuscation (Athalye et al., 2018)). However, this does not mean they are less robust.
Alternatively, as DBI suggests, LeNet and other simple networks might have achieved relatively good representations but without high accuracy.
To enable visualization of this metric, we plotted in Figure 3 a projection into two dimensions of all the points in the decision space of unknown classes.
All this is done while preserving the high-dimensional distance between the points.
Here we use the Isomap (Tenenbaum et al., 2000) to achieve this effect.
It can be easily observed that CapsNet’s results for unknown classes are more clustered and thus form a better-deﬁned cluster than other architectures.
4. E XPERIMENTS ON A MALGAM M ETRIC
In this section, the AM for all the networks is evaluated, which is based on the similarity of soft-labels for networks that were trained in all classes.
The results shown in Table 3 reveal almost the same representation ranking as the robustness rankings related to L 2 .
Interestingly, although both DBI and
AM differ widely in concept and calculation procedure, the rankings are both similar and close to the
L 2 .
This further suggests that both metrics agree on what would be a good representation quality and can be used to evaluate representation in newer methods.
A visualization as well as the Pearson correlation of the metric is included in the supplementary works.
4. P ARTS OF A W HOLE
In (Agarwal et al., 2019), adding a loss to force features to be close to the feature centroid was shown to be beneﬁcial against adversarial attacks. This is consistent with the proposed DBI metric, demonstrating that both soft-label space as well as other feature spaces beneﬁt from projections to nearby positions.
At the same time, we further support the existence of a trade-off between accuracy and robustness for deeper DNNs ﬁrst point out in (Tsipras et al., 2019).
Other types of networks, such as CapsNet seem to avoid it to some extent.
Therefore, the trade-off is shown to vary with the architecture and computing dynamics of a model.
Lastly, we hypothesize that a representation bias may hold in which features learned are only invariant to the dataset and not unseen classes.
C ONCLUSIONS
Here we proposed the Raw Zero-Shot method to evaluate the representation of classiﬁers.
In order to score the soft-labels, two metrics were formally deﬁned based on a different hypothesis of representation quality.
Results suggest that the evaluation of the representation of both metrics (DBI and AM) are linked with the robustness of neural networks.
In other words, quickly attacked neural networks have a lower representation score.
Interestingly, LeNet scores well in both metrics, albeit being the least accurate.
LeNet is followed up by AllConv and NIN, which are less complicated/profound than other models which suggest that deeper architectures might be trading-off representation quality for accuracy.
These results shown here further support the claim that there is a trade-off between accuracy and robustness in current deep learning (Tsipras et al., 2019).
Thus, the proposed Raw Zero-Shot was able to evaluate the representation quality of state-of-the-art
DNNs and show their shortcomings concerning adversarial attacks, explaining many of the current problems.
It also opens up new possibilities for both the evaluation (i.e. as a quality assessment) and the development (e.g., as a loss function) of neural networks.
S UPPLEMENTARY W ORK A T HE P OSSIBILITY OF A R EPRESENTATION B IAS
The experiments show the possibility of representations to not work well for unknown classes. Many of these classes, however, share similar representations such as the dog and cat, truck and car. Thus, here we formulate a possible interpretation for the results. The objective of a supervised learning algorithm is perhaps to map the input to output in such a way that the decision boundary reﬂects the real decision boundary.
To achieve this, it is known that when dealing with complex problems, learning algorithms need ﬁrst to learn a set of invariant features that are present throughout classes such that their recognition becomes robust against variations in the dataset. Human beings, however, learn a set of invariant features that is not only able to solve current tasks or recognize current classes. We learn a set of features that can describe most if not all unseen classes and unknown tasks.
Thus, we deﬁne representation bias as the bias towards invariant features that describe current seen classes or tasks but fail to describe unknown classes and tasks.
B E XTENDED A NALYSIS O F DBI M ETRIC
Figure 4 shows a visualization of DBI’s results with t-Distributed Stochastic Neighbour Embedding (t-SNE) (Maaten & Hinton, 2008).
DBI results are visualized using a projection into two dimensions while focusing on neighbour distances. The idea for having this visualisation is to investigate whether The ﬁgure 4 shows that CapsNet forms pretty dense projections. Even if we form clusters to have different classes, the gaps between the classes will be too low relatively to other architectures.
It also shows that in the high-dimensional space all the logits are moderately close to each other. While for the other architectures, there exists some points which can form their own cluster and be termed as different label. Hence for these architectures it can have one or more different classes for the same label which is contradicting to our hypothesis that there should exist only a single class for a single label.
This inference can also be linked to adversarial attacks and supports the argument that CapsNet is relatively more robust than the other architectures.
To enable a visualisation of the metric, the computed histograms ( h’ i and h i from Equation 5) are plotted in Figure 5. It is interesting to note that in Figure 5, the histograms from CapsNet are different from the other ones, by the entirely different architecture employed by CapsNet. This reveals that this metric can capture such representation differences.
Table 5 reveals that Amalgam metric is less reliable than DBI. Its p-values are also high enough to render its relationship less obvious. Recall, however, that the ranking based on Amalgam is very similar to the raking obtained with DBI metric.
[Equation]
where
[Equation] [Table]
The Amalgam Metric showed that both CapsNet and AllConv have the best scores which is in accordance with their top robustness score.
Figure 6 shows a visualization of equation 6 which is part of the main equation of Amalgam Metric.
On analysing further the phenomena about absolute difference between h’ and h in the Amalgam Metric or D in equation 6.
It can be noted from the ﬁgure that for most labels of CapsNet and AllConv the difference is relatively low than the other architectures.
This contributes to have CapsNet and AllConv to have best scores.
Further investigations can be carried out to analyse the effect of a label in adversarial attack based on this ﬁgure.
This can also provide insight on the labels which are robust to adversarial attacks.
A further study can also be carried out to analyse the characteristics of the neural network’s representation which makes a label more robust than other labels.