S IMILARITY OF N EURAL A RCHITECTURES B ASED ON I NPUT G RADIENT T RANSFERABILITY
Anonymous authors
Paper under double-blind review
A BSTRACT
In this paper, we aim to design a quantitative similarity function between two neural architectures.
Specifically, we define a model similarity using input gradient transferability.
We generate adversarial samples of two networks and measure the average accuracy of the networks on adversarial samples of each other.
If two networks are highly correlated, then the attack transferability will be high, resulting in high similarity.
Using the similarity score, we investigate two topics: (1) Which network component contributes to the model diversity?
(2) How does model diversity affect practical scenarios?
We answer the first question by providing feature importance analysis and clustering analysis.
The second question is validated by two different scenarios: model ensemble and knowledge distillation.
We conduct a large-scale analysis on 69 state-of-the-art ImageNet classifiers.
Our findings show that model diversity takes a key role when interacting with different neural architectures.
For example, we found that more diversity leads to better ensemble performance.
We also observe that the relationship between teacher and student networks and distillation performance depends on the choice of the base architecture of the teacher and student networks.
We expect our analysis tool helps a high-level understanding of differences between various neural architectures as well as practical guidance when using multiple architectures.
I NTRODUCTION
The last couple of decades have seen the great success of deep neural networks (DNNs) in real-world applications, e.g. , image classification (He et al., 2016a) and natural language processing (Vaswani et al., 2017).
The advances in the DNN architecture design have taken a key role in this success by making the learning process easier ( e.g. , normalization methods (Ioffe & Szegedy, 2015; Wu &
He, 2018; Ba et al., 2016) or skip connection (He et al., 2016a)) enforcing human inductive bias in the architecture ( e.g. , convolutional neural networks (CNNs) (Krizhevsky et al., 2012; Simonyan & Zisserman, 2015)) or increasing model capability with the self-attention mechanism ( e.g. , Transformers (Vaswani et al., 2017)).
With different design principles and architectural elements, there have been proposed a number of different neural architectures; however, designing a distinguishable architecture is expensive and needs heavy expertise.
One of the reasons for the difficulty is that there is only a little knowledge of the difference between two different neural architectures.
Especially, if one can quantify the similarity between two models, then we can measure which design component actually contributes significantly to diverse properties of neural networks.
We also can utilize the quantity for new model design ( e.g. , by neural architecture search (NAS)
(Zoph & Le, 2017)).
In this paper, we aim to define the similarity between two networks to quantify the difference and diversity between neural architectures.
Existing studies have focused on dissecting each network component layer-by-layer (Kornblith et al., 2019; Raghu et al., 2021) or providing a high-level understanding by visualization of loss surface (Dinh et al., 2017), input gradient (Springenberg et al., 2015; Smilkov et al., 2017), or decision boundary (Somepalli et al., 2022).
On the other hand, we aim to design an architecture agnostic and quantitative score to measure the difference between the two architectures.
We especially focus on the input gradients, a widely-used framework to understand model behavior, e.g., how a model will change predictions by local pixel changes (Sung, 1998;
Simonyan et al., 2014; Springenberg et al., 2015; Smilkov et al., 2017; Sundararajan et al., 2017;
Bansal et al., 2020; Choe et al., 2022).
If two models are similar, then the input gradients are similar.
However, because an input gradient is very noisy, directly measuring the difference between input gradients is also very noisy.
Instead, we use the adversarial attack transferability as the proxy measure of the difference between input gradients of two networks.
Consider two models A and B , and the input x .
We generate adversarial samples x A and x B to A and B , respectively.
Then we measure the accuracy of B for x A (acc A → B ) and vice versa.
If A and B are similar and assume an optimal adversary, then acc
A → B will be almost zero, while if A and B have distinct input gradients, then acc
A → B will not be dropped significantly.
We also note that adversarial attack transferability will provide a high-level understanding of the difference between model decision boundaries (Karimi &
Tang, 2020).
We define a model similarity based on attack transferability and analyze the existing neural architectures, e.g. , which network component affects the model diversity the most?
We measure the pairwise attack transferability-based network similarity of 69 different neural architectures trained on ImageNet (Russakovsky et al., 2015), provided by Wightman (2019).
Our work is the first extensive study for model similarity on a large number of state-of-the-art ImageNet models.
Our first goal is to understand the effect of each network module on model diversity.
We first choose 13 basic components ( e.g. , normalization, activation, the design choice for stem layers) that consist of neural architectures and list the components of 69 networks.
For example, we represent ResNet (He et al., 2016a) as f ResNet =
[ Base architecture =
CNN , Norm = BN , Activation = ReLU , . .
We analyze the contribution of each network component to the model diversity by using the feature importance analysis by gradient boosting regression on model similarities, and the clustering analysis on model similarities.
Our analyses show that the choice of base architecture ( e.g. , CNN,
Transformer) contributes most to the network diversity.
Interestingly, our analysis shows that the design choice for the input-level layers ( e.g. , stem layer design choice) determines the network diversity as much as the choice of core modules ( e.g. , normalization layers, activation functions).
Our study is not only limited to the understanding of component-level architecture design, but we also focus on analyzing the effect of model diversity in practical scenarios.
Particularly, we measure the model ensemble performances by controlling the diversity of the candidate models, e.g. , ensemble “similar” or “dissimilar” models by our similarity score.
Here, we observe that when we ensemble more dissimilar models, the ensemble accuracy gets better; more diversity leads to a better ensemble performance .
We observe that the diversity caused by different initialization, different hyper-parameter choice, and different training regimes is not as significant as the diversity caused by architecture change.
We also observe that the ensemble of models from the same cluster performs worse than the ensemble of models from different clusters.
Similarly, by choosing more diverse models, the number of wrong samples by all models is getting decreased.
Our findings confirm that the previous study conducted in simple linear classifiers (Kuncheva & Whitaker, 2003) is also aligned with recent complex large-scale neural networks.
As our third contribution, we provide a practical guideline for the choice of a teacher network for knowledge distillation (KD) (Hinton et al., 2015).
We train 25 distilled ViT-Ti models with diverse teacher networks.
Interestingly, our findings show that the performance of the distilled model is highly correlated to the similarity between the teacher and the student networks rather than the accuracy of the teacher network; if the student networks and the teacher network are based on the same architecture ( e.g. , both are based on Transformer), then a similar teacher provides better knowledge; if the student and teacher networks are based on different architectures ( e.g. , Transformer and
CNN), then we observe that choosing a more dissimilar teacher will lead to a better distillation performance.
Our findings are partially aligned with previous KD studies, i.e. , using a similar teacher network leads to a better KD performance (Jin et al., 2019; Mirzadeh et al., 2020).
However, the existing studies only focus on the scenario when both teacher and student networks are based on the same architecture, exactly aligned with our findings.
Our observation extends previous knowledge to the case of when the teacher and student networks significantly differ ( e.g. , Transformer and CNN).
N ETWORK SIMILARITY BY INPUT GRADIENT TRANSFERABILITY
In this section, we propose a similarity measure between two networks based on adversarial attack transferability.
Our interest lies in developing a practical toolbox to measure the architectural difference between the two models quantitatively.
Existing studies for a similarity between deep neural networks have focused on comparing intermediate features (Kornblith et al., 2019; Raghu et al., 2021), understanding loss landscapes (Dinh et al., 2017; Li et al., 2018; Park & Kim, 2022), or decision boundary (Somepalli et al., 2022).
However, their approaches cannot measure the distance between two different networks (Li et al., 2018; Kornblith et al., 2019) or need expensive computation costs for re-training networks multiple times (Somepalli et al., 2022).
There also has been another line of research based on prediction-based statistics, e.g. , comparing the wrong and correct predictions (Kuncheva & Whitaker, 2003; Geirhos et al., 2018; 2020; Scimeca et al., 2022).
Although this approach can be used for measuring a model diversity (Kuncheva & Whitaker, 2003), they only can capture the prediction itself, not the change of prediction.
However, just focusing on prediction values could be misleading.
Because recent deep models are highly complex, the change of predictions (or sensitivity) is more practical than focusing on the prediction itself.
For example,
Geirhos et al.
(2020) showed that recent complex deep neural networks show highly similar predictions.
Also, a deep model easily changes its prediction with a neglected small noise ( e.g. , adversarial attack), and the perturbation varies by different models, as shown in our experiments.
In this paper, we focus on how two models change their predictions by input changes, e.g. , the input gradient.
Input gradient is a popular visualization tool for understanding DNNs by highlighting where a model attends (Sung, 1998; Simonyan et al., 2014).
An input gradient-based method has several advantages; it is computationally efficient, and no additional training is required; it can provide a visual understanding of the given input.
Usually, an input gradient-based quantity measures how the input gradient of a model matches the actual foreground (Choe et al., 2022), i.e. , it usually measures how a model explains the input well.
Because we are interested in comparing two different models, we focus on how input gradients are similar between them.
Especially, we generate an adversarial sample using a model and test the adversarial sample to the other architecture.
Adversarial attack transferability is also a good approximation of measuring the difference between model decision boundaries (Karimi & Tang, 2020).
Comparing the decision boundaries of two models will provide a high-level understanding of how two models behave differently for input changes.
Unfortunately, an exact decision boundary is not achievable, and an expensive and inexact sampling approximation is required (Somepalli et al., 2022).
In this paper, we analyze the difference between two decision boundaries by generating an adversarial sample for each decision boundary.
If two models have similar decision boundaries, then the adversarial samples will be transferred ( i.e. , will have high adversarial attack transferability); if two models have dissimilar decision boundaries, then the adversarial samples will not be transferred.
The detailed discussion can be found in Appendix H.
Formally, we generate adversarial samples x
A and x B of model
A and B for the given input x .
Then, we measure the accuracy of model
A using the adversarial sample for model B (called acc B → A ).
If A and B are the same, then acc B
→ A will be zero (for simplicity, here, we may assume that the adversary can fool a model perfectly).
On the other hand, if the input gradients of A and B differ significantly, then the performance drop will be neglectable because the adversarial sample is made almost similar to the original image ( i.e. , ∥ x − x B ∥ ≤ ε ).
Let X AB be the set of inputs where both
A and B predict correctly, y be the ground truth label, and I ( · ) be the indicator function.
We use the following quantity to measure the similarity between two models:
[Equation]
where ε s is a small scalar value to prevent the ill-posedness of s ( · ) .
If A = B and we have an oracle adversary, then s ( A, A )
= log 100 .
In practice, a strong adversary ( e.g. , PGD (Madry et al., 2018) or AutoAttack (Croce & Hein, 2020)) can easily achieve a nearly-zero accuracy if a model is not trained by an adversarial attack-aware strategy, e.g. , adversarial training (Madry et al., 2018; Cohen et al., 2019).
On the other hand, if the adversarial attacks on A are not transferable to B and vice versa, then s ( A, B ) = log ε s .
We add more discussions how equation 1 can serve as a similarity score function from a functional point of view in Appendix A.
Now, we provide an extensive analysis of a large number of state-of-the-art networks using our quantity.
Our analysis focuses on two questions.
(1) Which network component contributes to the diversity between models?
(2) How the model diversity affects the practical applications, e.g. , model ensemble or knowledge distillation?
Note that several studies attempted to answer the second question in different ways (Kuncheva & Whitaker, 2003; Teney et al., 2022), but our work is the first work to analyze 69 state-of-the-art ImageNet models based on model similarity.
As observed by previous works in a limited number of models without considering the change of architecture and large-scale training datasets, our findings show that diversity can contribute to practical applications in the state-of-the-art complex deep neural networks trained on large-scale datasets.
[Table]
Settings.
We use the projected gradient descent (PGD) for the adversary.
We set the iteration to 50, the learning rate to 0.
[Table]
and ε to 8/255 for the PGD attack.
We also test other attack methods, such as AutoAttack and PatchFool (Fu et al., 2022) in Appendix G.
We select 69 neural architectures trained on ImageNet (Russakovsky et al., 2015) from the PyTorch Image Models library (Wightman, 2019) 1 .
Notice that to reduce the unexpected effect of a significant accuracy gap, the chosen model candidates are limited to the models whose top-1 accuracy is between 79% and 83%.
We also ignore the models with unusual training techniques, such as training on extra training datasets, using a small or large model input resolution ( e.g. , less than 200 or larger than 300), or knowledge distillation.
When A and B take different input resolutions, then we resize the attacked image from the source network for the target network.
We also sub-sample 10% ImageNet validation images ( i.e. , 5, images) to measure the similarity.
This strategy makes our similarity score more computationally efficient.
The full pairwise similarities of 69 models can be found in Appendix D.
M ODEL ANALYSIS BY NETWORK SIMILARITY
We analyze how each model component affects the diversity between networks quantitatively by feature importance analysis and clustering analysis (Sec. 3.1).
We also show that the diversity caused by different training strategies is not as significant as caused by different neural architectures (Sec. 3.2).
3. W HICH ARCHITECTURAL COMPONENT CAUSES THE DIFFERENCE
MOST
[Figure]
Settings.
We list 13 components of each architecture, e.g. , normalization ( e.g. , BN (Ioffe & Szegedy, 2015) and LN (Ba et al., 2016)), activation functions ( e.g. , ReLU (Krizhevsky et al., 2012) and GeLU (Ramachandran et al., 2017))
the existence of depthwise convolution, or stem layer ( e.g. , 7 × 7 conv, 3 × 3 conv, or 16 × 16 conv with stride 16 – a.k.a.
“patchify” stem (Liu et al., 2022)).
The list of sub-modules is shown in
Tab. 1 and Appendix B. We then convert each architecture as a feature vector based on the listed submodules.
For example, we convert
ResNet as f ResNet =
[ Base arch =
CNN , Norm = BN , Activation =
ReLU , . . . ] .
Feature important analysis.
The full list of components of 69 architectures is in Appendix B.
Now, we measure the feature importance by fitting a gradient boosting regressor (Friedman, 2001) on the feature difference ( e.g. , f ResNet-50 − f DeiT-base ) measured by Hamming distance and the corresponding similarity ( e.g. , 2.3).
The details of the regressor are in Appendix B.
We use the permutation importance (Breiman, 2001); We measure how the trained regression model changes the predicted similarity by randomly changing each feature ( e.g. , normalization).
The feature importance of each architectural component is shown in Fig.
We first observe that the choice of base architecture ( e.g. , CNN (Krizhevsky et al., 2012), Transformer (Vaswani et al., 2017), and MLPMixer (Tolstikhin et al., 2021)) contributes to the similarity most significantly.
Fig.
1 also shows that the design choice of the input layer ( i.e. , stem layer design choice or input resolution) affects the similarity as much as the choice of basic components such as normalization layers, activation functions, and the existence of attention layers.
On the other hand, we observe that the modified efficiency-aware convolution operations, such as depth-wise convolution (Chollet, 2017) or group convolution (Xie et al., 2017), are not effective for model diversity.
Clustering analysis.
We additionally provide a clustering analysis based on the architecture similarities.
We construct a pairwise similarity graph A between all 69 architectures where its vertex denotes an architecture, and its edge denotes the similarity between two networks.
We perform the spectral clustering algorithm (Ng et al., 2001) on A where the number of clusters K is set to 10: We compute the Laplacian matrix of A , L = D − A where D is the diagonal matrix and its i -th component is (cid:80)
j A ij .
Then, we perform K-means clustering on the K -largest eigenvectors of L , where
K is the number of clusters ( i.e. , K = 10 ).
More details of spectral clustering are in Appendix C.
Tab. 2 shows the clustering results on 69 networks and the top-5 keywords for each cluster based on term frequency-inverse document frequency (TF-IDF) analysis.
Specifically, we treat each model feature (Tab. 1) as a word and compute TF and IDF by treating each architecture as a document.
Then we compute the average TF-IDF for each cluster and report top-5 keywords.
Similar to Fig.
1, the base architecture ( e.g. , CNN in Cluster 5, 6, 10 and Transformer in Cluster 2, 3) and the design choice for the stem layer ( e.g. , Cluster 1, 2, 4, 5, 6, 7, 8, 10) repeatedly appear at the top keywords.
Especially, we can observe that the differences in base architecture significantly cause the diversity in model similarities, e.g. , non-hierarchical Transformers (Cluster 1), hierarchical networks with the patchified stem (Cluster 2), hierarchical Transformers (Cluster 3), CNNs with 2D self-attention (Cluster 4, 5), ResNet-based architectures (Cluster 6), and NAS-based architectures (Cluster 7).
3. T RAINING STRATEGY AND SIMILARITY SCORE
The architectural difference is not the only cause of the model diversity.
We compare the effect by different architecture choices ( e.g. , ResNet and Vision Transformer) and by different training strategies, while fixing the model architecture, as follows:
Different initializations can affect the model training by the nature of the stochasticity of the training procedure.
For example, Somepalli et al.
(2022) showed that the decision boundary of each architecture can vary by different initializations.
We also consider different optimization hyper-parameters ( e.g. , learning rate, weight decay).
Finally, we study the effect of different training regimes ( e.g. , augmentations, type of supervision).
For example, the choice of data augmentation ( e.g. , Mixup (Zhang et al., 2018), or CutMix (Yun et al., 2019)) or the optimization techniques ( e.g. , (Szegedy et al., 2016)) can theoretically or empirically affect adversarial robustness (Zhang et al., 2021;
Park et al., 2022; Shafahi et al., 2019;
Chun et al., 2019).
Similarly, we investigate the effect of different types of supervision, such as selfsupervision ( e.g. , MOCO (Chen et al., 2021c), MAE (He et al., 2022) and BYOL (Grill et al., 2020)) or semi-weakly supervised learning (Yalniz et al., 2019).
Settings.
Different initializations: We train 21 ResNet-50 models and 16 ViT-S from scratch individually by initializing each network with different random seeds.
Different hyper-parameters:
We train 28 ResNet-50 models and 9 ViT-S by choosing different learning rates, weight decays, and learning rate schedulers.
The full list of hyper-parameters is shown in Appendix E. Different training regimes: We collect 23 ResNet-50 models and 7 ViT-S models with different training
[Table]
regimes.
22 ResNet-50 models include 1 model trained by PyTorch (Paszke et al., 2019), 4 models trained by GluonCV
(Guo et al., 2020), a semi-supervised and semi-weakly supervised models on billion-scale unlabeled images (Yalniz et al., 2019), 5 models trained by different augmentations (Cutout (DeVries & Taylor, 2017), Mixup (Zhang et al., 2018), manifold Mixup (Verma et al., 2019), CutMix (Yun et al., 2019), and feature CutMix); 11 optimized models by Wightman et al.
7 ViT-S models contain the original ViT training setup (Dosovitskiy et al., 2021), a stronger data augmentation setup (Touvron et al., 2021), the training setup with distillation (Touvron et al., 2021), an improved DeiT training setup (Touvron et al., 2022b), and self-supervised training fashions including MoCo v3 (Chen et al., 2021c), MAE (He et al., 2022) and BYOL (Grill et al., 2020).
Tab.
D.1 shows the comparison of similarity scores between the same architecture but different learning methods (a smaller similarity means more diversity).
First, we observe that using different random initialization or different optimization hyper-parameters shows high correlations with each other (almost ≥ 4.2) while the average similarity score between various neural architectures is 2.73.
In other words, the difference in initializations or optimization hyper-parameters does not significantly contribute to the model diversity.
Second, we observe that using different learning technique ( e.g. , different augmentation methods, and different types of supervision) remarkably affect the similarity score (3.27 for ResNet and 3.44 for ViT), but is not as significant as the architectural difference (2.73).
Furthermore, we discover that the change of similarity caused by different initializations or hyper-parameters is less marked than the change caused by different architecture (Fig.
D.1).
In practice, learning the same network with significantly different learning techniques is challenging and time-consuming because the hyper-parameter search space is tremendously large (Bergstra &
Bengio, 2012).
Especially, a novel optimization technique ( e.g. , using more data points for semi- or self-supervised learning, using different augmentation methods, or adopting advanced optimization settings) sometimes needs a completely different optimization setting compared to the vanilla setting.
On the other hand, learning a new architecture is easier than using a different learning paradigm in terms of hyper-parameter selection and implementation.
Thus, we can conclude that if we need multiple diverse models, then using multiple architectures is more effective and easier to optimize.
P RACTICAL APPLICATIONS OF THE NETWORK SIMILARITY SCORE
In this section, we show the practical usage of our similarity score.
First, we show that using more diverse models will lead to better ensemble performance (Sec. 4.1).
Second, we suggest a similaritybased guideline for the choice of a teacher model when distilling to a specific architecture.
Different from several existing studies that show a more similar teacher is better, our findings show that their observation is valid only if the teacher and student networks are based on the same architecture ( e.g. , Transformer).
On the other hand, when the teacher and student networks are based on different architectures, a more dissimilar teacher is better (Sec. 4.2).
4. M ODEL DIVERSITY AND ENSEMBLE PERFORMANCE
4.0 Similarity
E rr o r
R e d u c t
i
o n R a t e ( % )
The model ensemble is a practical technique to achieve high performances in practice, however, there is still not enough study on the choice of the models for the ensemble in large-scale complex models.
Previous studies are mainly conducted on very small datasets and linear models (Kuncheva & Whitaker, 2003).
We report the relationship between the model similarity and the ensemble performance based on the unweighted average method (Ju et al., 2018)
( i.e. , averaging the logit values of the ensembled models).
Because the ensemble performance is sensitive to the original model performances, we define Error Reduction Rate (ERR) as 1 −
Err ens ( M )| M | (cid:80) m ∈ M Err ( m ) , where M is the set of the ensembled models,
ERR( m ) denotes the top-1 ImageNet validation error of model m , and Err ens ( · ) denotes the top-1 error of the ensemble.
We first measure the 2-ensemble performances among the 69 neural architectures in Tab. 2 ( i.e. , the number of ensembles is (cid:0) 692 (cid:1) = 2346 ).
We plot the relationship between the model similarity and ERR in Fig.
We observe that there exists a strong negative correlation between the model similarity and the ensemble performance (Pearson correlation coefficient − 0 .
31 with p-value 1 .
74 × 10 − 54 ), i.e. , more diversity leads to better ensemble performance .
[Figure]
diversity by architecture difference and ensemble performance.
We report ensemble performances by varying the number of ensembled models ( N ) and the diversity of the ensembled models.
The diversity is controlled by choosing the models from k different clusters (larger k denotes more diversity)
We also conduct N -ensemble experiments with N ≥ 2 based on the clustering results in Tab. 2.
We evaluate the average ERR of the ensemble of models from k clusters, i.e. , if N = 5 and k = 3 , the ensembled models are only sampled from the selected 3 clusters while ignoring the other 7 clusters.
We investigate the effect of model diversity and ensemble performance by examining k = 1 . . .
N ( i.e. , larger k denotes more diverse ensembled models).
We report three evaluation metrics in Fig.
ImageNet top-1 error of the ensemble (lower is better), error reduction rate (ERR) (higher is better), and the number of wrong samples by all ensembled models (lower is better).
In all metrics, we observe that the ensemble of more diverse models ( i.e. , by increasing the number of clusters for the model selection) shows better ensemble performance.
Interestingly, in Fig.
3a, we observe that when the number of clusters for the model selection ( k ) is decreased, the ensemble performance by the number of ensembled models ( N ) quickly reaches saturation.
Similarly, in Fig.
3c, we observe that the number of wrong samples by all models is decreased by selecting more diverse models.
In other words, the different architecture choices can lead to different model decision boundaries.
[Figure]
are shown.
“All” means N = k in Fig.
3 ( i.e. , # of ensembled models =
# of Clusters).
In Sec. 3.2, we showed that the different training strategies are not as effective as different architectures in terms of diversity.
To examine the effect of this observation on the ensemble scenario, we report the ensemble results of different training strategies, i.e. , the same ResNetand ViT-S models in Tab.
D.1.
For comparison with different architectures, we also report the ensemble of different architectures where all ensembled models are from different clusters ( i.e. , when N = k in
Fig. 3).
Fig.
4 shows that although using diverse training regimes (blue lines) improves ensemble performance most among other training techniques (red and green lines), the improvements by using different architectures (yellow lines) are more significant than the improvements by using different training regimes (blue lines) with large gaps.
4. M ODEL DIVERSITY AND KNOWLEDGE DISTILLATION
Knowledge distillation (Hinton et al., 2015) is a training method for transferring rich knowledge of a well-trained teacher network.
Intuitively, distillation performance affects a lot by choice of the teacher network, however, the relationship between similarity and distillation performance has not yet been explored enough, especially for ViT.
In this subsection, we investigate how the similarity between teacher and student networks contributes to the distillation performance.
There are several studies showing two contradictory conclusions; Jin et al.
Mirzadeh et al.
(2020) showed that a similar teacher leads to better distillation performance; Touvron et al.
(2021) reports that distillation from a substantially different teacher is beneficial for ViT training.
Settings.
We use the same distillation setting by Touvron et al.
; We train ViT-Ti models with the hard distillation strategy (Hinton et al., 2015) ( i.e. , using the predicted label by the teacher as the ground truth label).
We train 25 ViT-Ti models with different teacher networks from Tab. 2.
Here, we include all non-ViT teachers that have a similarity score larger than 2.5, and 25 models among 67 models (except for ViT and DeiT) are sampled according to the distribution of the similarity scores to prevent sampling bias.
We describe the detailed training settings in Appendix F.1.
Fig.
5a illustrates the relationship between the teacher-student similarity and the distillation performance.
Fig.
5a tends to show a not significant negative correlation between teacher-student similarity and distillation performance ( − 0.32 Pearson correlation coefficient with 0.12 p-value).
However, if we only focus on when the teacher and student networks are based on the same architecture ( i.e. ,
Transformer), we can observe a strong positive correlation (Fig. 5b) – 0.70 Pearson correlation coefficient with 0.078 p-value.
In this case, our observation is aligned with Jin et al.
; Mirzadeh et al.
(2020): a teacher similar to the student improves distillation performance.
However, when the teacher and student networks are based on different architectures ( e.g. , CNN), then we can observe a stronger negative correlation (Fig. 5c) with − 0.51 Pearson correlation coefficient and 0.030 pvalue.
In this case, a more dissimilar teacher leads to better distillation performance.
We also test
[Figure] [Figure]
other factors that can affect distillation performance in Appendix F.2; We observe that distillation performance is not correlated to teacher accuracy in our experiments.
Why we observe contradictory results for Transformer teachers (Fig. 5b) and other teachers (Fig. 5c)?
Here, we conjecture that when the teacher and student networks differ significantly, distillation works as a strong regularizer.
In this case, using a more dissimilar teacher can be considered as a stronger regularizer (Fig. 5c).
On the other hand, we conjecture that if two networks are similar, then distillation works as easy-to-follow supervision for the student network.
In this case, we can assume that a more similar teacher will work better because a more similar teacher will provide more easy-to-follow supervision for the student network (Fig. 5b).
In our experiments, we found that the regularization effect improves distillation performance better than easy-to-follow supervision ( i.e. , the best performing distillation result is by a CNN teacher).
Therefore, in practice, we recommend using a significantly different teacher network for achieving better distillation performance ( e.g. , using RegNet (Radosavovic et al., 2020) teacher for ViT student as Touvron et al.
L IMITATIONS AND DISCUSSIONS
We include three discussions related to our similarity score.
Due to the space limitation, the full details of each item can be found in
Appendix G. Our discussions include:
• Robustness of our similarity score to the choice of adversarial attack method.
We show that the similarity rankings by PGD (Madry et al., 2018), AutoAttack (Croce & Hein, 2020), and PatchFool (Fu et al., 2022) do not differ significantly.
• How adversarially trained models affect our similarity scores.
• An efficient approximation of our analysis for a novel model ( i
e. , for a novel 70th model for our analysis).
We show that instead of generating adversarial attacks for all 70 models, just using the attacked images from the novel model could be enough.
• Other possible applications of our similarity score ( e
g. , a novel architecture development, analyzing the hyper-parameter sensitivity, model selection for selecting diverse models).
C ONCLUSION
We have explored similarities between image classification models to investigate what makes the model diverse and whether developing and using diverse models are required.
For quantitative and model-agnostic assessment of the similarity, we have suggested a new score based on attack transferability demonstrating differences in input gradients.
Using our new similarity function, we conduct a large-scale and extensive analysis using 69 state-of-the-art ImageNet models.
We have shown that macroscopic architectural properties, such as base architecture and stem architecture, have a greater impact on similarity than microscopic operations, such as types of used convolution, with numerical analysis.
Furthermore, the effectiveness of the training strategy is minor compared to model architectural design-related features.
Finally, we have firstly discovered the advantages of using diverse models in ensemble or distillation strategies for a practical scenario with a large-scale training dataset and a highly complex architecture.
A PPENDIX
We include additional materials in this document, such as the details of our similarity function (A) the network details (B), spectral clustering method (C), training settings (E), and setting of knowledge distillation (F.1).
We also provide experimental results of computing pairwise similarities (D) and distillation performance (F.2).
A M ORE DETAILS OF OUR SIMILARITY FUNCTION
Ideally, we would like to define a similarity function between two models with the following three properties: (1) x = arg min y d ( x, y ) (2) d ( x, y ) =
d ( y, x ) (3) d ( x, y ) >
d ( x, x ) if x ̸ = y .
The adversarial attack transferability generally satisfies these properties.
If we assume the adversary is optimal, then the transferred attacked accuracy (acc B → A ) will be zero and it will be the minimum (because accuracy is non-negative).
acc
A → B + acc B
→ A is symmetric.
It will satisfy d ( x, y ) ≥ d ( x, x ) if x ̸ = y where it is a weaker condition than (3).
In other words, adversarial attack transferability and equation 1 can serve as a similarity score function from a functional point of view.
B D ETAILS OF NETWORKS USED FOR OUR ANALYSIS
We use 69 models in our research to evaluate the similarity between models and to investigate the impact of model diversity.
In the main paper, we mark the names of models based on their research paper and PyTorch Image Models library ( timm ; 0.6.7 version) (Wightman, 2019).
Tab.
B.1 shows the full list of the models based on their research paper and timm alias.
In Tab.
B.2 and Tab.
B.3, We show the full network specification defined by Tab. 1.
We follow the corresponding paper and timm library to list the model features.
We fit a gradient boosting regressor (Friedman, 2001) based on Scikit-learn (Pedregosa et al., 2011) and report the permutation importance of each component in Fig.
The number of boosting stages, maximum depth, minimum number of samples, and learning rate are set to 500, 12,
[Table]
and 0.02, respectively.
Permutation importance is computed by permuting a feature 10 times
[Table] [Table]
C S PECTRAL CLUSTERING DETAILS
We use the normalized Laplacian matrix to compute the Laplacian matrix.
We also run K-means clustering 100 times and choose the clustering result with the best final objective function to reduce the randomness by the K-means clustering algorithm.
We visualize the pairwise distances of the spectral features ( i.e. , K -largest eigenvectors of L ) ofarchitectures in Fig.
C.1.
Note that rows and columns of Fig.
C.1 are sorted by the clustering results.
[Table]
[Table] [Table]
[Figure]
Fig.
C.1 shows block diagonal patterns, i.e. , in-cluster similarities are large while between-cluster similarities are small.
D P AIRWISE SIMILARITIES
Fig.
D.1 shows that the similarities between different architectures (Fig.
D.1a and the off-diagonal elements of Fig.
D.1b) are smaller than the similarities within the same architecture with different initializations or hyper-parameters.
Fig.
D.2 indicates the pairwise similarity among 69 models.
We can observe a weak block pattern around clusters, as also revealed in Fig.
C.1.
E T RAINING SETTING DETAILS FOR S EC . 3.
We train 21 ResNet-50 models and 16 ViT-S from scratch individually by initializing each network with different random seeds.
We further train 28 ResNet-50 models by randomly choosing learning rate ( × 0.1, × 0.2, × 0.5, × 1, × 2, and × 5 where the base learning rate is 0.1), weight de-
[Table]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure] [Table]
[Figure]
[Table]
[Figure]
[Table] [Figure] [Table] [Figure]
[Table]
[Figure]
[Table] [Figure] [Table] [Figure]
[Table]
[Figure]
m _ 24 _ p 16 _ c s p r e s n e t4.
[Figure]
Similarity between different networks.
ResNetViT-S
ResNetViT-S 4.
1.0 Base Init
Base
Init
Base
Init
LR
WD
LR
WD
LR WD Base Init LR WD
[Figure]
[Figure] [Figure]
cay ( × 0.1, × 0.2, × 0.5, × 1, × 2, and × 5 where the base weight decay is 1e-4), and learning rate scheduler (step decay or cosine decay).
Similarly, we train 9 ViT-S models by randomly choosing learning rate ( × 0.2, × 0.4, and × 1 where the base learning rate is 5e-4) and weight decay ( × 0.2, × 0.4, and × 1 where the base weight decay is 0.05).
Note that the DeiT training is unstable when we use a larger learning rate or weight decay than the base values.
Finally, we collect 22 ResNetmodels with different training regimes: 1 model with standard training by PyTorch (Paszke et al., 2019); 4 models trained by GluonCV
(Guo et al., 2020) 2 ; a semi-supervised model and semi-weakly supervised model on billion-scale unlabeled images by Yalniz et al.
(2019) 3 ; 5 models trained by different augmentation methods (Cutout (DeVries & Taylor, 2017), Mixup (Zhang et al., 2018), manifold Mixup (Verma et al., 2019), CutMix (Yun et al., 2019), and feature CutMix 4 ; 10 optimized ResNet models by (Wightman et al., 2021) 5 .
We also collect 7 ViT-S models with different training regimes, including the original ViT training setup (Dosovitskiy et al., 2021) 6 , a stronger data augmentation setup in the Deit paper (Touvron et al., 2021)-3 6 , the training setup with distillation (Touvron et al., 2021)-3 6 , an improved DeiT training setup (Touvron et al., 2022b)-3 6 , and
We weights provided by https://github.com/rwightman/ pytorch-image-models/releases/tag/v0.1-rsb-weights
[Table]
[Figure]
self-supervised training fashions by MoCo v3 (Chen et al., 2021c) 7 , MAE (He et al., 2022)
and
BYOL (Grill et al., 2020) 9 .
We do not use adversarially-trained networks because the adversarial training usually drops the standard accuracy significantly (Tsipras et al., 2019).
F K NOWLEDGE DISTILLATION F. T RAINING SETTING DETAILS FOR S EC . 4.
We train ViT-Ti student models with 25 different teacher models using hard distillation strategy.
We follow the distillation training setting of DeiT official repo 10 , only changing the teacher model.
Note that we resize the input images to the input size the teacher model requires if the input sizes of student and teacher models differ.
If a teacher model needs a different input resolution, such as 240 × 240 and 256 × 256 , we resize the input image for distilling
[Figure]
it.
Because DeiT-Ti has low classification accuracy compared to teacher models, the similarity score is calculated between DeiT-S and 25 models.
The 25 teacher models are as follows:
BoTNet-26 , CoaT-Lite Small , ConViT-B , ConvNeXt-B , CrossViT-B , CSPDarkNet-53 ,
CSPResNeXt-50 , DLA-X-102 , DPN-107 , EfficientNet-B2 , FBNetV3-G , GC-ResNet-50 , gMLP-S , HaloRegNetZ , MixNet-XL , NFNet-L0 , PiT-S , RegNetY-032 , ResMLP-24 , ResNet-50 ,
SEHaloNet33 , Swin-T , TNT-S , VisFormer-S , and XCiT-T12 .
F. T EACHER ACCURACY AND DISTILLATION PERFORMANCE
The similarity between teacher and student networks may not be the only factor contributing to distillation.
For example, a stronger teacher can lead to better distillation performance (Yun et al., 2021).
In Fig.
F.1, we observe that if the range of the teacher accuracy is not significantly large enough ( e.g. , between 79.5 and 82.5), then the correlation between teacher network accuracy and distillation performance is not significant; 0.049 Pearson correlation coefficient with 0.82 p-value.
In this case, we can conclude that the teacher and student networks similarity contributes more to the distillation performance than the teacher performance.
G L IMTATIONS AND DISCUSSIONS
Robustness to the choice of adversarial attack methods.
Our similarity score is based on attack transferability using the representative gradient-based attack method, PGD attack (Madry et al.
To explore the effect of the choice of attack methods, we compute similarity scores on two different gradient-based attack methods; Autoattack (Croce & Hein (2020)), which is the state-of-the-art method attack method based on the ensemble of four diverse attacks, and Patchfool attack (Fu et al.
(2022)), which is specially designed for vision transformers.
We sample 8 models among 69 models for testing the effect of Autoattack on the similarity score: ViT-S , CoaT-Lite Small , ResNet-101 , LamHaloBotNet50 , ReXNet-150 ,
NFNet-L0 , Swin-T and Twins-pcpvt .
Figure G.1a shows the high correlation between similarity scores using PGD and Autoattack; it shows a correlation coefficient of 0.98 with a p-value of 1 . 43 × 10 − 18 .
For testing the Patchfool attack, we only generate adversarial perturbations on ViT-S and get attack transferability to all other models (68 models) because Patchfool targets transformerbased models.
In the results, Patchfool also shows a high correlation compared to the PGD attack (Pearson correlation coefficient 0.91 with p-value 3 .
62 × 10 − 27 ).
The results show that our similarity score is robust to the choice of adversarial attack methods if the attack is strong enough.
Note that
PatchFool needs a heavy modification on the model code to extract attention layer outputs manually.
On the other hand, PGD and AutoAttack are model-agnostic and need no modification on the model code.
Therefore, if the PatchFool attack and the PGD attack show almost similar similarity rankings, we would like to recommend using PGD because it is easier to use for any architecture. [Figure]
Similarity scores for adversarially trained networks.
Because our method is based on an adversarial attack, it could be affected by an adversarial attack-aware learning method.
We first note that our analysis is invariant to the adversarial training; as shown by Tsipras et al.
(2019), adversarial robustness and accuracy are in a trade-off, and there is no ImageNet model with accuracy aligned with our target models yet.
Also, our model similarity is based on the difference in the change of the model predictions.
As shown by Tsipras et al.
and Ilyas et al.
(2019), adversarial training will lead to a different decision boundary.
Therefore, we think that although our score can be affected by adversarial training, our score is still valid for adversarially trained models because an adversarially trained model is actually dissimilar to other models in terms of the decision boundary and representation learning.
An efficient approximation for a new model.
We can use our toolbox for designing a new model; we can measure the similarity between a novel network and existing N architectures; a novel network can be assigned to clusters (Tab. 2) to understand how it works.
However, it requires generating adversarial samples for all N +1 models ( e.g. , 70 in our case), which is computationally inefficient.
Instead, we propose the approximation of equation 1 by omitting to compute the accuracy of the novel network on the adversarial samples of the existing networks.
It will break the symmetricity of our score, but we found that the approximated score and the original score have high similarity (0.
Pearson correlation coefficient with almost 0 p-value) as shown in Figure G.2).
As an example, we tested Gluon-ResNeXt-50 (Guo et al., 2020), which has the same architecture with ResNeXt-50 (Xie et al., 2017) in 69 models, and the distilled version of DeiT-S model (Touvron et al., 2021).
Because the difference is not significant as we observed in Tab.
D.1 and
Fig.
4, we expect that Gluon-ResNeXt-50 is assigned to the same cluster with ResNeXt-50 , and distilled DeiT-S is assigned to the same cluster with DeiT-S .
We observe that, as we expected, each network is assigned to the desired cluster.
Therefore, we suggest using our efficient approximation for analyzing a novel network with our analysis toolbox.
Other possible applications.
We can use our similarity to develop a novel neural architecture or training technique by measuring how the proposed method can learn different representations compared to existing methods.
Our similarity also can be used for analyzing the hyper-parameter sensitivity.
Instead of reporting the performance changes by different hyper-parameters, one can report the average similarity of the trained models with different hyper-parameters.
If the average similarity is significantly low, then we can argue that the method is sensitive to hyper-parameter selection.
We expect this scenario is especially useful for the task, sensitive to the hyper-parameter selection (Choe et al., 2022; Gulrajani & Lopez-Paz, 2021).
Finally, our similarity can be used for the model selection criteria for selecting diverse models, e.g. , constructing a new dataset with diverse pre-trained models for annotation candidates (Kuznetsova et al., 2020; Chun et al., 2022). [Figure]
H A DVERSARIAL ATTACK TRANSFERABILITY AND DECISION BOUNDARY
Adversarial attack transferability is highly related to the decision boundaries of models.
More specifically, measuring adversarial attack transferability is a good approximation of measuring the difference between model decision boundaries Karimi & Tang (2020).
Comparing decision boundaries of two models will provide a high-level understanding of how two models behave differently for the changes of input.
Unfortunately, an exact decision boundary is not achievable and an expensive and inexact sampling approximation is required (Somepalli et al., 2022).
For example, Somepalli et al.
(2022) uses a sampling-based approximation with 2,500 samples.
More specifically, Somepalli et al.
(2022) generates 2,500 perturbed samples (expanded by a coordinate defined by a randomly chosen triplet of inputs) for a given input and measures the ratio of “agreement” of two models ( e.g. , if both two models predict 1,250 perturbed samples to the same label, then the similarity of decision boundaries for the given input is 1250/2500 = 0.5).
However, it needs an expensive sampling approximation procedure and the approximation bound of the sampling method is not guaranteed.
On the other hand, we can analyze the difference between two decision boundaries by generating an adversarial sample for each decision boundary.
If two models have similar decision boundaries, then the adversarial samples will be transferred ( i.e. ,, will have high adversarial attack transferability); if two models have dissimilar decision boundaries, then the adversarial samples will not be transferred.
Compared to the sampling-based approximation, adversarial attack transferability is more efficient (we used 50 iterations for the attack, while sampling needs 2,500 samples) and less stochastic than a sampling-based approximation.
Also, we employed an adversarial attack based on gradient descent where the convergence of gradient descent is theoretically guaranteed, while sampling approximation has no guarantee on its approximation error.
Therefore, adversarial attack transferability can work as a good proxy measure of the difference of model decision boundaries.