T RAINING I NTERPRETABLE C ONVOLUTIONAL N EURAL N ETWORKS TOWARDS C LASS -S PECIFIC F ILTERS
Anonymous authors
Paper under double-blind review
A BSTRACT
Convolutional neural networks (CNNs) have often been treated as “black-box” and successfully used in a range of tasks.
However, CNNs still suffer from the problem of ﬁlter ambiguity – an intricate many-to-many correspondence relationship between ﬁlters and features, which undermines the models’ interpretability.
To interpret CNNs, most existing works attempt to interpret a pre-trained model, while neglecting to reduce the ﬁlter ambiguity hidden behind.
To this end, we propose a simple but effective strategy for training interpretable CNNs.
Specifically, we propose a novel Label Sensitive Gate (LSG) structure to enable the model to learn disentangled ﬁlters in a supervised manner, in which redundant channels experience a periodical shutdown as ﬂowing through a learnable gate varying with input labels.
To reduce redundant ﬁlters during training, LSG is constrained with a sparsity regularization.
In this way, such training strategy imposes each ﬁlter’s attention to just one or few classes, namely class-speciﬁc.
Extensive experiments demonstrate the fabulous performance of our method in generating sparse and highly class-related representation of the input.
Moreover, comparing to the standard training strategy, our model displays less redundancy and stronger interpretability.
I NTRODUCTION
Convolutional Neural Networks (CNNs) demonstrate extraordinary performance in various visual tasks (Krizhevsky et al., 2012;
He et al., 2016; Girshick, 2015; He et al., 2017a).
However, the strong expressive power of CNNs is still far from interpretable, which signiﬁcantly limits its applications that require humans’ trust or interaction, e.g. self-driving and medical image analysis (Caruana et al., 2015; Bojarski et al., 2017).
In this paper, we argue that ﬁlter ambiguity is one of the most critical reasons that hampers the interpretability of CNNs.
As a matter of fact, previous studies has shown that 1) ﬁlters in CNNs generally extract features of a mixture of various semantic concepts, including objects, parts, scenes, textures, materials and colors (Zhang et al., 2018b; Bau et al., 2017); and that 2) there is also redundant overlap between features extracted by different ﬁlters (Prakash et al., 2019).
The intricate many-to-many correspondence relationship between ﬁlters and features is so-called ﬁlter ambiguity as shown on the left of Figure 1.
Obviously, in high convolutional layers which might capture class-related feature, ﬁlter ambiguity contradicts our intention of an interpretable CNN, because it hinders humans from interpreting the concepts of a ﬁlter (Zhang et al., 2018b), which has been shown as an essential role in the visualization and analysis of networks (Olah et al., 2018) in human-machine collaborative systems (Zhang et al., 2017a;c).
Moreover, the unnecessary overlap between features extracted by different ﬁlters leads to under-utilization of a model’s expressiveness (Prakash et al., 2019)
Therefore, reducing ﬁlter ambiguity is critical to obtain better feature with better interpretability and less redundancy.
However, it is non-trivial to achieve such a goal barricaded by substantial challenges.
First, most interpretability-related research simply focuses on post-hoc interpretation of ﬁlters (Szegedy et al., 2013;
Bau et al., 2017), which manages to interpret the main semantic concepts captured by a ﬁlter but fails to alleviate the ﬁlter ambiguity prevalent in pretrained models.
Second, many existing works such as VAEs’ variants (Higgins et al., 2017; Burgess et al., 2018; Kim & Mnih, 2018; Chen et al., 2018; Kumar et al., 2017) and InfoGAN (Chen et al., 2016) try to disentangle data representation and obtain better interpretability in an unsupervised way. However, it is proved that unsupervised dog, cat, ship cat, plane plane, dog, ship cat, dog ship, cat, dog dog, ship t op c onvo l u ti on a l f ilt e r s activation maps often activated by ambiguous filters normal CNN ship activation maps often activated by class-specific filters our CNN dog cat plane t op c onvo l u ti on a l f ilt e r s class-related representation detecting adversarial samples improving localization Car Trunk Plane
LSG CNN Predicted ( Ours )
Cat Cat Cat
Dog STD CNN Predicted True Labels Bird advantages in applications many-to-many learning on disentangled features without inductive bias is impossible (Locatello et al., 2018), which challenges the works above.
Considering the aforementioned challenges, we shed light on enforcing the one-class for one-ﬁlter relationship during training (instead of post-hoc) in a supervised manner only with classiﬁcation labels.
To this end, we propose a novel training strategy that coerces each ﬁlter into extracting features from only one or few classes for classiﬁcation tasks, namely disentangling ﬁlters towards classspeciﬁc .
Speciﬁcally, we design a Label Sensitive Gate (LSG) structure on the top of convolutional ﬁlters, which limits each ﬁlter’s activation only to its speciﬁc input label(s).
In our training process, we periodically insert LSG into the CNN and jointly minimize the classiﬁcation cross-entropy and the sparsity of LSG, so as to keep the model’s performance on classiﬁcation and meanwhile encourage class-speciﬁc ﬁlters.
Experiments proved the LGS ﬁlter yield better representation for images hence leading to promising applications. Our training method makes data representation sparse and highly correlated with the labeled class, which not only illustrates the alleviation of ﬁlter ambiguity but also enhances the interpretability of the network.
The advantages of our method are concretely substantiated by sparser correlation between ﬁlters, sparser ﬁlter-class correlation, better explanation for misclassiﬁcation and more precise localization of labeled objects.
Contributions
The contributions of this work can be summarized as: (1) we propose a novel training strategy for CNNs which forces each ﬁlter to extract features mainly from only one or a few classes; (2) moreover, we propose a metric to evaluate ﬁlter ambiguity, which can also be used as a regularization in our training to encourage class-speciﬁc ﬁlters; (3) ﬁnally, our training enables ﬁlters to output sparse and class-related representation, which helps alleviate ﬁlter redundancy, improve interpretability of the model.
R ELATED W ORKS
Existing works related to our work include post-hoc ﬁlter interpretation, learning disentangled representation and model pruning.
Post-hoc Interpretation for Filters is widely studied, which aims to interpret the patterns captured by ﬁlters in pretrained CNNs.
Plenty of works visualize the pattern of a neuron as an image, which is the gradient (Zeiler & Fergus, 2014; Mahendran & Vedaldi, 2015; Simonyan et al., 2013) or accumulated gradient (Mordvintsev et al., 2015;
Olah et al., 2018) of a certain score about the activation of the neuron.
Bau et al.
(2017) determine the main visual patterns extracted by a convolutional ﬁlter by treating it as a pattern detector.
Some other works transfer the representation in CNN into an explanatory graph (Zhang et al., 2017b; 2018a) or a decision tree (Zhang et al., 2019), which aim to ﬁgure out the visual patterns of ﬁlters and the relationship between co-activated patterns.
Posthoc ﬁlter interpretation helps to understand the main patterns of a ﬁlter but makes no change to the existing ﬁlter ambiguity of the pretrained models, while our work aims to train interpretable models.
Learning Disentangled Representation refers to learning data representation that encodes different semantic information into different dimensions.
As a principle, it proves impossible to learn disentangled representation without inductive bias (Locatello et al., 2018).
Unsupervised methods such as variants of VAEs (Kingma & Welling, 2014) and InfoGAN (Chen et al., 2016) rely on regularization.
VAEs (Kingma & Welling, 2014) are modiﬁed into many variants (Higgins et al., 2017;
Burgess et al., 2018; Kim & Mnih, 2018;
Chen et al., 2018; Kumar et al., 2017), while their disentangling performance is sensitive to hyperparameters and random seeds.
InfoGAN (Chen et al., 2016) learns disentangled hidden representations by maximizing the mutual information between input and generated images.
Some other unsupervised methods rely on special network architectures including interpretable CNNs (Zhang et al., 2018b) and CapsNet (Sabour et al., 2017).
As for supervised methods, Thomas et al.
(2018) propose to disentangle with interaction with the environment; Bouchacourt et al.
(2018) apply weak supervision from grouping information, while our work applies weak supervision from classiﬁcation labels.
Model Pruning reduces structure redundancy and therefore is widely used for both model efﬁciency and better generalization, while our work uses it to alleviate ﬁlter ambiguity and improve interpretability.
Many existing methods prune pretrained CNNs for better efﬁciency without significant deterioration in performance, including channel-wise pruning (He et al., 2017b;
Molchanov et al., 2016; Liu et al., 2017) and ﬁlter-wise pruning (Luo et al., 2017).
The methods mentioned imply that CNNs trained by standard training strategy tend to have redundant ﬁlters.
Further research exploits redundant ﬁlters to improve generalization and suppress over-ﬁtting by temporally pruning during training: Dense-Sparse-Dense (Han et al., 2016) encourages sparse weights by regularization and then recovers dense weights by removing the regularization; RePr (Prakash et al., 2019) repeatedly prunes and reinitializes redundant ﬁlters to reduce ﬁlter overlap.
M ETHOD
Learning disentangled ﬁlters in CNNs alleviates ﬁlter ambiguity and meanwhile narrows the gap between human’s perception and CNN’s representations.
In this section, we ﬁrst present an ideal case of class-speciﬁc ﬁlters, which is a direction for our training, and then we elaborate on our method about how to train an interpretable network.
[Figure]
a normal CNN extracts a mixture of features from different classes (Zhang et al
2018b), which is a symptom of ﬁlter ambiguity.
In contrast, when ﬁlters are ideally class-speciﬁc, each ﬁlter extracts features mainly from only one class.
Given the labeled class of an input, the ﬁlters irrelevant to the class have weak activation, and hence classiﬁcation performance changes little when they are shut down.
3. C LASS -S PECIFIC F ILTERS
For CNNs in classiﬁcation tasks, it is demonstrated that the ﬁlters in the last convolutional layer extract high-level features related to certain classes more or less (Zeiler & Fergus, 2014).
We suppose the most ideal case, as shown in Figure 2, is that each ﬁlter extracts features mainly from only one class, i.e. each ﬁlter is mainly relevant to one class.
We call such ﬁlters ideally class-speciﬁc.
Obviously, given the labeled class of an input, the classiﬁcation performance changes little if we shut down the channels from the ﬁlters irrelevant to the class.
This is the direction for our training method elaborated later.
To have a rigorous deﬁnition of “ideally class-speciﬁc”, for a CNN, we use a matrix G ∈ { 0 , 1 } C × K to measure the relevance between ﬁlters and classes, where K is the number of ﬁlters, C is the number of classes, and each element G kc represents the relevance between the k -th ﬁlter and c -th class, as shown in Figure 3.
The k -th ﬁlter extracts features mainly in the c -th class when and only when G kc = 1 .
Given the label y ∈ { 1 , 2 , ..., C } of an input, we can index a row G y ∈ { 0 , 1 } K from the matrix G , which can be used as a gate multiplied to the feature maps to shut down those irrelevant channels.
We call ﬁlters in a CNN as ideally class-speciﬁc ﬁlters, when the network’s prediction ˜
y G approaches that of original network ˜
y
after the feature maps from the last convolutional layer are multiplied by the gate G y . [Figure]
3. P ROBLEM FORMULATION
In order to train a CNN towards disentangling ﬁlters to class-speciﬁc and meanwhile keep the original classiﬁcation performance, as opposed to the standard training scheme that entangles ﬁlters, we introduce a Label-Sensitive Gate (LSG) path in addition to the standard path of forward propagation.
In such a LSG path, some channels are shut down by multiplication with a learnable gate.
This path’s classiﬁcation performance is regarded as a regularization for disentanglement training.
As shown in Figure 3, let us denote θ as the network parameters.
The network forward propagates in two paths: 1) the standard (STD) path predicting ˜
y θ , and 2) the LSG path with gate matrix G predicting ˜ y Gθ where the learnable gate G y for the input labeled with y is multiplied to the feature map before the linear layer.
Initially, the elements in the gate matrix G for a CNN are unknown.
Intuitively, we can search for them by exploring the binary space and ﬁnd one solution that yields the best classiﬁcation performance through LSG path, i.e., to solve Φ 0 ( θ ) =
min G : G k is one-hot CE ( y || ˜
y Gθ ) , where the constraint on G k applies to ∀ k ∈ { 1 , 2 , ..., K } and Φ 0 evaluates the performance of the CNN with ﬁlters disentangled.
Therefore, it is natural to add Φ 0 into training loss as a regularization that forces ﬁlters to be class-speciﬁc.
Thus, we get the following formulation of the original problem to train a CNN towards ideally class-speciﬁc ﬁlters as
[Equation]
However, the problem is difﬁcult to solve in practice.
On the on hand, the assumption that each ﬁlter is ideally disentangled to extract only one class hardly holds, since it is usual for several classes to share one high-level feature in CNNs; on the other hand, binary vectors in a non-continuous space are difﬁcult to optimize with gradient descent.
To address these two issues, we relax the constraints from two perspectives.
First, we relax the one-hot vector G k to sparse vector by L 1 regularization (cid:107) G (cid:107) 1 where at least one element equals to( (cid:13)(cid:13)
G k (cid:13)(cid:13) ∞
= 1 )
Since || G k || ∞ = 1 ensures (cid:107) G (cid:107) 1 to be larger than K , we actually apply (cid:12)(cid:12) (cid:107) G (cid:107) 1 − s [Equation] as the sparsity regularization term in order to make the norm of gate matrix close to a constant target norm s ≥ K .
Second, we relax the binary vector G k ∈ { 0 , 1 } C to a continuous vector G k ∈ [0 , 1] C .
Therefore Φ 0 now becomes
[Equation] [Equation]
where a is a coefﬁcient to balance classiﬁcation and sparsity.
Φ can be regarded as a metric of ﬁlter ambiguity , which means a CNN with higher class-speciﬁcity of ﬁlters will have a lower Φ (contrary to common sense, here we expect Φ to be as small as possible).
Replacing Φ 0 in Equation 1 with Φ , we get an intermediate problem min θ ( CE ( y || ˜
y
θ )
+ λ Φ( θ )) .
It is mathematically equivalent if we move min G within Φ to the left most (see Appendix A for proof).
Thus, we formulate a relaxed problem as
[Equation]
The relaxed problem is easier to solve by jointly optimizing θ and G with gradient descent, compared to either the discrete optimization in the original problem, or the nested optimization in the intermediate problem.
Solving the relaxed problem, we can obtain a CNN for classiﬁcation with class-speciﬁc ﬁlters, where G precisely describes the correlation between ﬁlters and classes.
3. O PTIMIZATION
To solve the optimization problem formulated in Equation 3 with gradient descent, we alternately optimize G and θ to improve the classiﬁcation performance of the model while ensuring the constraints of sparsity on G , as is shown in Algorithm 1 1 .
In this scheme, we run the LSG path to update both G and θ in some epochs of a period, and run the STD path to update θ in the other epochs.
When G is updated with gradient, G k will be normalized by (cid:13)(cid:13)
G k (cid:13)(cid:13) ∞ to ensure (cid:13)(cid:13) G k (cid:13)(cid:13) ∞
= 1 , and then clipped into range [0 , 1] .
After solving the relaxed problem, we can further tighten the constraint on LSG to G ∈ { 0 , 1 } C × K for stronger sparsity of G .
In this work, we tighten into a simple and special case, where some ﬁlters are related to only one class (i.e. G k is one-hot) for class-speciﬁc features, and the other ﬁlters are related to all classes (i.e. G k is all-one) for class-sharing features.
Speciﬁcally, if G k has only one element G kc ≥ h (threshold h ∈ (0 , 1] ) G k will be set to one-hot, otherwise G k will be set to all-one. Algorithm
1: for E in epochs do 2: for N in batches do 3: if E%3 =
= 1 then 4: ˜
y Gθ ← prediction through the LSG path with G 5:
Cost ← λ ( CE ( y || ˜
y Gθ )
+ a (cid:12)(cid:12)(cid:13)(cid:13) G (cid:13)(cid:13) 1 − s (cid:12)(cid:12) ) 6:
G is updated using the gradient decent as G ← G − (cid:15) ∂Cost∂G ; 7:
Each column of G k is normalized as G k ←
G k (cid:107) G k (cid:107) ∞ ; 8:
G ← clip ( G, 0 , 1) 9: else 10: ˜ y
θ ← prediction through the STD path 11:
Cost ← CE ( y || ˜
y θ ) 12: end if 13: θ ← θ − (cid:15)
∂Cost∂θ 14: end for 15: end for
E XPERIMENT
In this section, we conduct ﬁve experiments. We ﬁrst delve into LSG training from three aspects, so as to respectively study the effectiveness of LSG training, the class-speciﬁcity of ﬁlters and the correlation between ﬁlters train with LSG.
After that, we demonstrate LSG’s application on improving object localization and adversarial samples detection.
In the following parts, we denote our training method Label-Sensitive Gate as LSG , the standard training as STD , and CNNs trained with them as
LSG CNNs and STD CNNs , respectively.
Training setting
We trained ResNet20s (He et al., 2016) on CIFAR20 and ResNet152s (He et al., 2016) on PASCAL VOC 2010 (Everingham et al.) on classiﬁcation task with LSG/STD training. ResNet20s are evaluated in this 4.1, 4.2, 4.3 and 4.5, and ResNet152s are evaluated in 4.1 and 4.4.
For fair comparison, all the code is implemented on the Pytorch framework and tested on GTX 1080Ti GPUs.
We report the validation performance on last epoch following common practice.
For ResNet20s, the default settings include: batch size is 256; the optimizer is SGD with momentum of 0.9 (Sutskever et al., 2013); the initial learning rate is 0.1; and the total training epochs is 150.
The ResNet152s are ﬁnetuned from model pretrained on ImageNet (Deng et al., 2009). Parameters are frozen except the top 2 bottleneck blocks, gate matrix and linear layers. The setting is: batch size is 32; the optimizer is Adam (Kingma & Ba, 2014); the initial learning rate is 1e-5 for STD path and 1e-3 for LSG path; the total training epochs is 150. Beside, we preprocess PASCAL VOC to be a classiﬁcation dataset for training ResNet152s: we crop out images for the objects in 6 classes (bird, cat, dog, cow, horse and sheep) and resize the image to 128x128; then randomly reassign 3644 objects for training and 1700 objects for testing.
No segmentation label is used in training ResNet152s. 4. E FFECTIVENESS OF LSG T RAINING To begin with, we conduct experiments to demonstrate the effectiveness of our LSG training in learning a sparse gate matrix and yielding a low Φ that evaluates class-speciﬁcity of ﬁlters.
Basic Quantitative Evaluation
To compare the performance on classiﬁcation and ﬁlter disentanglement of the STD CNN and the LSG CNN, we calculate their test accuracy, cross entropy, L Norm (cid:107) G (cid:107) 1 and Φ as mentioned in Section 3.2.
As shown in Table 1, LSG CNN’s even slightly outperforms STD CNN in test accuracy, and cross entropy of both is comparable, while LSG CNN has much lower L1 Norm and Φ .
The L1 Norm and the Φ indicate LSG learns class-speciﬁc ﬁlters that hence reduce ﬁlter ambiguity.
These metrics quantitatively demonstrate LSG’s capability of learning a sparse gate matrix and disentangling ﬁlters without any sacriﬁce on classiﬁcation accuracy.
[Table]
Visualization of Label-Sensitive Gate Matrix To illustrate the relevance between classes and the learned ﬁlters described in the gate matrix, we visualize gate matrices in Figue 4.
Subﬁgure (a) indicates that the LSG training yields a sparse LSG matrix where each ﬁlter is only related to one or few classes.
Subﬁgure (b) comes from a LSG training without the sparsity regularization to the gate matrix (L1 norm), from which we observe ﬁlter ambiguity where a ﬁlter would extract ambiguous features from multiple classes.
Accordingly, LSG training effectively learn sparse gate matrix, and this characteristic originates from our sparsity regularization – the L1 norm on the gate
[Figure] [Figure]
4. C LASS -S PECIFIC F ILTERS
To substantiate that LSG training learns class-speciﬁc ﬁlters and reveal how it works, we explore the
LSG CNN with ﬁlters masked for certain class(es) from which we observe the similarity between feature vectors and gates. The experiments are conducted on the ResNet20 trained in 4.1.
Masking Filters for Certain Class
Although the sparse gate matrix implies ﬁlters are classspeciﬁc, to further verify the LSG training learns class-speciﬁc ﬁlters, we remove the ﬁlters highly related to certain class(es) referencing the gate matrix (i.e. G kc greater than a threshold), and then visualize the classiﬁcation confusion matrices.
As shown in Figure 5, when ﬁlters highly related to “plane” are removed, to our surprise, the LSG CNN fails to recognize the ﬁrst class ”plane”; nevertheless, the STD CNN still manages to recognize ”plane”.
Analogously, when masking the ﬁlters highly related to the ﬁrst and sixth classes, we observe similar phenomenon.
This demonstrates that in the LSG CNN, the features from a certain class are almost extracted by the speciﬁc ﬁlters highly related to the class, while other ﬁlters extract little feature from the class.
Therefore the ﬁlters are trained to be class-speciﬁc under the encouragement of the sparse gate matrix.
Similarity Between Features and Gates This experiment explains how the LSG matrix encourages class-speciﬁc ﬁlters by analyzing the similarity between feature from each class and each row in the LSG matrix.
In our models there is a global average pooling (GAP) layer after the last convolutional layer, which yields a feature vector denoted as a .
We use inner production to measure the similarity S cy between the feature vectors a y of images in the class y and the c -th row G c in the LSG matrix (see Appendix B for details).
We calculate S cy over all true positive (TP) and false negative (FN) images respectively and obtain similarity matrices S TP , S FN ∈ R C × C , as shown in Figure 6.
From the ﬁgure, we observe two phenomena and provide the following analysis.
1) TP similarity matrix is diagonally dominant signiﬁcantly, which reveals how LSG works in disentangling ﬁlters to class-speciﬁc: LSG forces ﬁlters to yield feature vectors whose direction approaches that of the gate vector for its related class.
FN similarity matrix is far from diagonally dominant.
Besides, two classes with shared features, such as car & truck and ship & plane, have high similarity in the
FN similarity matrix, which enlightens us that hard samples with ambiguous feature across classes tends to be misclassiﬁed in LSG models.
Thus, the mechanism of misclassiﬁcation in the LSG CNN is probably the features across classes extracted by the shared ﬁlters .
[Figure]
4. C ORRELATION B ETWEEN F ILTERS
To study the ﬁlter correlation in LSG CNNs
[Figure]
we train a CNN with the LSG under a tighter constraint
[Figure]
so as to conveniently analyze the correlation between ﬁlters for different classes.
Constraint Tightening
Before tightening the constraint on gate matrix G ∈ { 0 , 1 } C × K according to Section 3.3, we ﬁrst conduct a statistic analysis on 20 different LSG models to ﬁgure out how many ﬁlters each class monopolizes.
The result indicates that in CIFAR10, each class tends to monopolize 10% total ﬁlters (6 for ResNet20’s last convolutional layer), and 10% from all ﬁlters are shared by classes.
Inspired by this phenomenon, we tighten the constraint accordingly by manually setting a ﬁxed LSG matrix.
In the ﬁxed LSG matrix, for each row G c there are 6 ones from 6 columns of one-hot G k , and 4 extra columns G k are all-one (see Appendix C for illustration).
This provides a setup where each class monopolizes 6 ﬁlters and 4 extra ﬁlters are shared by all classes.
Models trained with this constraint naturally inherits all features of previous LSG models, and moreover, has better disentangled ﬁlters.
We evaluate the correlation between ﬁlters with the orthogonality of their weights.
We train an AlexNet (Krizhevsky et al., 2012) and a ResNet20 with the ﬁxed LSG matrix or with STD, and then calculate the correlation between ﬁlters in each models.
The correlation between two ﬁlters is deﬁned as the inner product of their normalized weights, visualized as matrices in Figure 7.
In subﬁgures (a), (c) for STD models, the ﬁlters are randomly correlated with each other.
On the contrary, subﬁgures (b), (d) for LSG models, the matrices are approximately block-diagonal, which means the correlation between the ﬁlters is limited into several ﬁlter groups corresponding to classes.
This implies that ﬁlters for the same class are highly correlated (nonorthogonal) for the co-occurrence of features extracted by them, while ﬁlters for different classes are almost uncorrelated (orthogonal) for the lack of co-occurrence. Through intuitive reasoning, we can explain why LSG training encourages ﬁlters for different classes to become orthogonal. Given a class c and a gate matrix, that assigns the ﬁlter k for class c and ﬁlter k (cid:48) for other class. During training, ﬁlter k (cid:48) is closed (i.e. its activation is masked) in the LSG path when class c ’s images input.
In order to ensure the STD path predict similar to LSG path, the ﬁlter k (cid:48) tends to be activated by class c as less as possible, which implies the weight of ﬁlter k (cid:48) is approximately perpendicular to V c , the linear space spanned by class c ’s features in a layer before.
The ﬁlter k for class c , however, tends to be activated by class c as saliently as possible so as to enable the CNN to recognize this class. So the weight of ﬁlter k is approximately within V c . Overall, the weights of ﬁlter k and ﬁlter k (cid:48) tends to be orthogonal. Filter Redundancy
With further analysis based on ﬁlter correlation, we discover that LSG reduce the redundancy of ﬁlters. For each element a i,j in the correlation matrix, we deﬁne that the i-th ﬁlter and j-th ﬁlter are correlated if a i,j > s where s is a varying threshold.
We count the ratio of elements in the correlation matrix satisfying the constrain and plot the results in subﬁgure (e).
It shows that LSG signiﬁcantly reduces the redundant weak (e.g. threshold = 0 . 3 ) correlation between most ﬁlter pairs, which are mostly for different classes. We explain the reduction of ﬁlter redundancy as a natural consequence of encouraging ﬁlters (for different classes) to be orthogonal.
For a set of ﬁlter groups orthogonal to each other, a ﬁlter in any group can not be a linear representation with the ﬁlers from other groups. This directly avoids redundant ﬁlter across groups. Besides, experiments in (Prakash et al., 2019) also verify the opinion that ﬁlter orthogonality reduces ﬁlter redundancy.
[Figure]
Class-Related Representation Based on ﬁlter correlation , we can even show that ﬁlters trained with LSG can yield highly class-related representation, namely the representation for an image tends to exactly correspond to its labeled class rather than to other classes. That’s based on a reasoning that in the highly activated channel of a representation, ﬁlters are less activated by other classes and less correlated to the ﬁlters for other classes. To verify this reasoning, we analyze the correlation between the ﬁlters highly activated by each class.
First we pick out each class c ’s dominant m ﬁlters, denoted as group A c (class c has highest average activation on them; m = 25 for AlexNet; m = 6 for RestNet20).
We deﬁne Inter-Class ﬁlter correlation as the average correlation between ﬁlters in different classes’ group: C IC = 1 C ( C − 1) m 2 (cid:80) c (cid:80) c (cid:54) =
c (cid:48) (cid:80) k ∈
A c (cid:80) k (cid:48)
∈
A c (cid:48) C k,k (cid:48) , where C k,k (cid:48) is the row k column k (cid:48) of the ﬁlter correlation matrix in Figure 7.
The results in Table 2 show that inter-class ﬁlter correlation in LSG is about half as much as that in STD, both on AlexNet and ResNet20.
This demonstrates that different classes tend to activate uncorrelated ﬁlters in LSG CNNs.
As a result, the representations for different classes have less overlap.
Thus we ﬁnally conﬁrm representation from LSG CNN is highly class-related.
[Table]
In this subsection, we exploit our class-speciﬁc ﬁlters to improve the localization of CNNs. LSG training encourages top convolutional each ﬁlter to focus on fewer classes, which obviously implies its feature map can localize a class better. We conducts experiments below to verify it.
Localization method
Resizing feature maps to input size is a widely used method to determine the area of objects or visual concepts, which not only works in localization task without bounding box labels (Bau et al., 2017), but also take an important role in network visualization and understanding the function of ﬁlters (Zhou et al., 2016).
We study LSG CNNs’ performs on localizing object classes with the feature map from both one ﬁlter and all ﬁlters.
For one ﬁlter, we bilinear interpolate its feature map and segment the region with values above the top 20% activation of the ﬁlter on the entire test dataset.
For all ﬁlers, we sums up their feature maps with the weights of the linear connections between each channels and an output class 2 .
By bilinear interpolating the sum feature map to input size, we get a classiﬁcation activation map (CAM) (Bau et al., 2017).
We segment the region with values above the top 20% activation in a CAM.
Thus we can get segmentation map for a class from feature maps.
Quantitative evaluation We train ResNet152s to do classiﬁcation on preprocessed PASCAL VOC and use Avg-IoU (average intersection over union) and AP15 (average precision 15%) to evaluate their localization for each class and all classes.
Higher metrics shows better localization.
Especially, for localization with one ﬁlter, we report Avg-IoU and AP15 averaged over all ﬁlters.
See Appendix E for detailed deﬁnition of the metrics.
The results for localization with one or all ﬁlters are shown in Table 3.
For localization with one ﬁlter, most classes is localized better with LSG CNN.
That’s because ﬁlters trained with LSG tends be activated by the labeled class rather than many other classes, which alleviates other classes’ interference on feature maps.
Therefore, as a weighted sum of better one-ﬁlter feature maps, LSG CNN’s CAMs also outperform LSG CNN’s.
[Table]
Visualiziation Besides the quantitative evaluation above
[Table]
in Figure
[Table]
we also visualize sample some images and their CAMs from the STD/LSG CNN.
We observe that the CAMs of STD CNN often activate extra or other semantic areas unrelated to the labeled class.
However, LSG training successfully helps the CNN ﬁnd a more precise area of the labeled objects in an image.
Such a phenomenon vividly demonstrates that LSG training improves the performance in localization.
[Figure]
4. A PPLICATION - A DVERSARIAL S AMPLE D ETECTION
This subsection shows an application of the highly class-related representation from our classspeciﬁc ﬁlters.
It is studied (Wang et al., 2018) that adversarial samples can be detected based on the anomalous behavior of their representation on each layers: in low layers of a neural network, the representation of an adversarial sample is similar to the original class, while on the high layers it is similar to the target class.
With the highly class-related representation from our LSG training, it will be much easier to distinguish which class the high-layer representation is similar to.
Therefore, we suppose this advantage will help in adversarial detection task.
To verify this judgment, we train a binary classiﬁer with the features of normal samples and adversarial samples extracted by global average pooling after each convolution layers of ResNet trained in 4.1.
As indicated in (Wang et al., 2018), the non-targeted adversarial samples result in semantic-closer class with the original class, which are hard to discriminate.
In this regard, we generate targeted adversarial samples by commonly used PGD (Madry et al., 2017) attack with (cid:15)
= 0 .and
iter = 40 and the adversarial target classes are from a random permutation of original classes besides each image’s true class.
In our experiments, we adopt random forest (Breiman, 2001) as the binary classiﬁer.
We randomly select 100, 500 and 1000 images for each class in CIFARdataset to form different sizes of training datasets while the test data is collected by randomly selecting 100 images for each class.
The experimental results are shown in Table 4.
We repeat each experiments ﬁve times and report the mean AUC scores.
The experimental results demonstrate that the class-related representation can better distinguish the abnormal behavior of adversarial samples which can help improve the robustness of the model.
[Table]
C ONCLUSION
In this work, we propose a simply yet effective structure – Label Sensitive Gate Matrix to disentangle the ﬁlters in CNNs.
With reasonable assumptions about the behaviors of ﬁlters, we derive regularization terms to constrain the form of the gate matrix.
As a result, the sparsity of the gate matrix encourages class-speciﬁc ﬁlters, and therefore yields sparse and highly class-related representations, which endows model with better interpretability.
We believe LSG is a promising architecture to disentangle ﬁlters in CNNs.
Referring to LSG’s successful utility and feasibility in classiﬁcation problem, we expect that LSG also have the potential to interpret other tasks like detection, segmentation etc, and networks more than CNNs, which is the direction of our future work.
A E QUIVALENCE P ROOF
This section is a supplementary proof for the equivalence (mentioned in 3.2) between
[Equation]
and
[Equation]
This equivalence is true referencing the Lemma below, if we set x ∈ X in the Lemma as θ ∈ R M , y ∈ Y as G ∈ { G ∈ [0 , 1] C × K } , f ( x ) as CE ( y || ˜ y θ ) , and g ( x, y ) as CE ( y || ˜
y Gθ )
+ a |(cid:107) G (cid:107) 1 − s |
[Equation]
Here, M is the number of parameters in a CNN, C is number of classes, K is the number of ﬁlters.
y is the label of an input, ˜
y
θ is the prediction of the CNN’s prediction through the STD path, and ˜ y
G θ is the prediction of the CNN’s prediction through the LSG path with G .
Lemma Given sets X, Y , and functions f : X (cid:55)→ R , g : X × Y
(cid:55)→ R .
f .
If f and g have lower bounds, then
[Equation]
Proof.
Given ∀ x ∈ X , f ( x ) is a constant if we take y as the only variable, so
[Equation]
Denote F ( x, y ) =
f ( x ) + g ( x, y ) , thus to prove equation 4 we only need to prove
[Equation]
This is obvious, because for ∀ x ∈ X, y ∈ Y ,
[Equation]
, so for ∀ x ∈ X
[Equation]
and then
[Equation]
Reversely, for ∀ x ∈ X, y ∈ Y , we have
[Equation]
so
[Equation]
B S IMILARITY B ETWEEN F EATURE V ECTORS AND G ATES
We design a similarity between the feature vector a y of a image in class y and the c -th row G c in the
LSG matrix.
Noticing a, G c ∈ R K , we use their inner production a Ty G c as a similarity in direction.
Thus, each pair of image in class y and class c will contribute an inner production a Ty G c to S cy – the average similarity between the feature vectors for class y and the c -th gate row.
Thus, we can further deﬁne the average similarity between the feature vectors for class y and the c -th gate row,
S cy
( D ) =
D test meana Ty G c : ( x, y ) ∈ D.
Here, D is a dataset to calculate the average on, and ( x, y ) is a pair of image and label in the dataset.
We can take D as all true positive or all false negative data, as conducted in 4.2.
C M ANUALLY F IXED G ATE M ATRIX
In 4.3 we use a manually ﬁxed gate matrix to train CNNs from scratch.
The gate matrix is visualized in Figure 9.
[Figure]
D C LUSTER CENTER EXPERIMENTS
Using the model with ﬁxed LSG mentioned in 4.3, we run k-means clustering on the feature vectors after the global average pooling in the STD CNN and the LSG CNN.
The dataset we used is CIFAR10.
We ﬁnd the LSG CNN yields better clustering centers, which is almost the same as the gate matrix we used (visualized in Figure 9), with ﬁlter groups reordered.
[Figure]
E M ETRICS FOR L OCALIZATION
This section gives detailed deﬁnition of the metrics we used in our localization experiments.
E. L OCALIZATION WITH O NE F ILTER
For an image x in class c (denoted as x ⊂ D c ∈ D , D is the dataset), we denote the ground-truth segmentation map for x as S x , and the segmentation map given by the resized feature map from ﬁlter k as ˆ S kx .
The ﬁlter k ∈ [1 , K ] , where K is the number of ﬁlers in the top convolutional layer.
The IoU (intersection over union) on this image is deﬁned as
[Equation]
The Avg-IoU (average intersection over union) for ﬁlter k to localize class c is deﬁned as
[Equation]
The AP15 (average precision 15%) for ﬁlter k to localize class c is deﬁned as
[Equation]
[Equation]
When c = arg max c (cid:48)
AP15 kc (cid:48) , we call ﬁlter k is for localizing class c , denoted as k ∈ F c , where F c is the set of ﬁlters for localizing class c .
The performance for ﬁlter k is evaluated with IoU k = IoU kc ∗ and AP15 k
= AP15 kc ∗ .
Thus we can deﬁne the Avg-IoU and AP15 for localizing class c as
[Equation]
and
[Equation]
Besides, the Avg-IoU and AP15 for localizing all classes is deﬁned as
[Equation]
and
[Equation]
E.
L OCALIZATION WITH
A LL F ILTERS
For an image x in class c (denoted as x ∈ D c ), we denote the ground-truth segmentation map for x as S x , and the segmentation map given by the resized classiﬁcation activation map (CAM) (Bau et al., 2017) as ˆ S x .
The IoU (intersection over union) on this image is deﬁned as
[Equation]
The Avg-IoU (average intersection over union) for localizing class c is deﬁned as
[Equation]
The AP15 (average precision 15%) for localizing class c is deﬁned as
[Equation]
Besides, the Avg-IoU and AP15 for localizing all classes is deﬁned as
[Equation]
and
[Equation]
= mean x ∈ D I { IoU x ) > 0 . 15 } .
F D EFENDING A DVERSARIAL S AMPLES
In this experiments, inspired by the highly class-related representation, we further explore LSG
CNNs’ potential in robustness for adversarial attacks.
Two black box attacks are conducted, including one pixel attack (Su et al., 2019) and local search attack (Narodytska & Kasiviswanathan, 2016).
They try to fool models according to the model’s predicted probability without access to the models’ parameters and architectures.
From the results shown in Table 5, we ﬁnd both the attacks gain attack success rates on STD CNN much higher than on LSG CNN.
This demonstrates that LSG training also improves robustness to CNNs.
We guess the robustness is caused by the increase of within-class distance and the decrease of between-class distance, which requires further veriﬁcation.
Robustness is another valuable characteristic of the highly class-related representation from the class-speciﬁc ﬁlters.
[Table]