Deconfounded Representation Similarity for Comparison of Neural Networks
Anonymous Author(s)
AffiliationAddress email
Abstract
Similarity metrics such as representational similarity analysis (RSA) and centeredkernel alignment (CKA) have been used to understand neural networks by comparing their layer-wise representations.
However, these metrics are confounded by thepopulation structure of data items in the input space, leading to inconsistent conclusions about the functional similarity between neural networks, such as spuriouslyhigh similarity of completely random neural networks and inconsistent domainrelations in transfer learning.
We introduce a simple and generally applicablefix to adjust for the confounder with covariate adjustment regression, which improves the ability of CKA and RSA to reveal functional similarity and also retainsthe intuitive invariance properties of the original similarity measures.
We showthat deconfounding the similarity metrics increases the resolution of detectingfunctionally similar neural networks across domains.
Moreover, in real-worldapplications, deconfounding improves the consistency between CKA and domainsimilarity in transfer learning, and increases correlation between CKA and modelout-of-distribution accuracy similarity.
Introduction
Deep neural networks (NNs) have achieved state-of-the-art performance on a wide range of machinelearning tasks by automatically learning feature representations from data [1, 2, 3, 4, 5].
However,these networks do not offer interpretable predictions on most applications and are seen as “blackboxes”.
It is thus crucial to understand the intricacies of neural networks before they are deployedon critical applications.
Previous work has made progress in understanding how a single neuralnetwork makes decisions with axiomatic attribution methods [6, 7] and understanding how multipleneural networks relate to each other with representation similarity measures [8].
Several similaritymeasures between representations have been proposed with different principles, including linearregression [9], canonical correlation analysis (CCA; [10, 11]), statistical shape analysis [12] andfunctional behaviors on down-stream tasks [13, 14, 15].
Another main-stream approach is based onrepresentational similarity analysis (RSA, [8, 16, 17, 18]) and centered kernel alignment (CKA, [19]),which compute the similarity between (dis)similarity matrices of two neural network representationson the same dataset.
RSA and CKA have been successfully applied to understand biological [20] and artificial NNs [21] bystudying the similarity of representations of different NNs in a single data domain.
However, we findthat the inter-example (dis)similarity matrices in the representation space of different NNs are highlycorrelated with a shared factor (i.e., a confounder): the (dis)similarity structure of the data items inthe input space, especially for shallow layers.
This confounding issue limits the ability of CKA toreveal similarity of models on the functional level [15], and can result in a spuriously high CKA evenbetween two random NNs.
Moreover, this can lead to counter-intuitive conclusions when using CKA deconfounding
[Figure]
dCKA
[Figure]
CKA and dCKA between two random ResNets ( NN 1, NN 2 ) deconfounding
CKA and dCKA between pretrained ( PT ) and fine-tuned ( FT ) ResNets
CKA=0.
dCKA=0.
[Figure]
to compare models trained in multiple domains and/or to deduce functional similarities of NNs, suchas in understanding transfer learning, multitask learning, and meta learning [22, 23, 24, 25, 26].
In this paper, we propose to adjust the representation similarity to improve its consistency w.r.t.
thefunctional similarity by regressing out the confounder, the inter-example (dis)similarity matrix inthe input space, from the (dis)similarity matrices of two representations, inspired by the covariateadjusted correlation analysis widely studied in biostatistics [27, 28].
This approach is simple andwidely applicable on any similarity measure built on the CKA and RSA framework.
Moreover, westudy the invariance properties of the deconfounded representation similarity and demonstrate itsbenefits on public image and natural language datasets with various NN architectures.
Overall, our contributions are:
• We study the confounding effect of the input inter-example similarity on the representationsimilarity between two NNs, which limits CKA and RSA from revealing consistent func-
tional similarities.
We propose a simple and generally applicable deconfounding fix anddiscuss the invariance properties of the deconfounded similarities.
• We verify that deconfounded similarities can detect functionally similar NNs from randomNNs, and small NN changes across domains where previous similarity measures fail.• We show that deconfounded similarities are more consistent with domain similarities in
transfer learning, on both image and language datasets, compared with existing methods.•
We demonstrate that deconfounded similarities on in-distribution datasets are more corre-
lated with out-of-distribution accuracy than the corresponding original similarities [15].
Preliminaries
2.1 Notation and prior work
Let X ∈ R n × p denote the input dataset with n datapoints and p features, and X m 1 f 1 ∈ R n × p 1 and
X m 2 f 2 ∈ R n × p 2 denote the m 1 th and m 2 th layer representations of two NNs of interest, f 1 ( X ) andf 2 ( X ) , with p 1 and p 2 nodes respectively.
We center and normalize the representation matrices byfirst removing the mean of each feature (i.e., each column), and then dividing by the Frobenius norm.
A standard approach for comparing representations of two NNs is to compare the similarity structuresin each network representation.
This can be done by first computing the similarity between every pairof examples in X m 1 f 1 and X m 2 f 2 with a similarity measure k ( · , · ) :
[Equation]
Here K m 1 f 1 , K m 2 f 2 ∈ R n × n are called representational similarity matrices (RSMs) 1 .
Second, anothersimilarity measure s ( · , · ) is applied to compare these two similarity structures, K m
[Equation]
and K
[Equation] [Equation]
This gives the similarity between the two representations.
The existing approaches vary by using different similarity measures for both levels of comparison.
CKA [19] employs a kernel function for the first level similarity k ( · , · ) , and a Hilbert-Schmidt
Independence Criterion (HSIC) estimator for the second, s ( · , · ) .
On the other hand, RSA-basedmethods use Euclidean distance to measure the inter-example dis similarity structure, and apply
Pearson’s correlation [8] or Spearman’s rank correlation [18] to quantify the similarity between twodissimilarity structures.
Although other approaches, e.g., linear regression [29] and CCA [10], havebeen proposed to compare NN representations, we focus on CKA and RSA in this paper due to theirwide usage in understanding the properties of NNs, such as transfer learning [22, 23, 30].
2.2 Illustration of the confounding in representation similarity
Given a dataset X with an inter-example similarity matrix K 0 =
k ( X, X ) in the input space, makingpredictions with NNs can be seen as modifying K 0 layer-by-layer, such that similarities between dataitems with different labels decrease and those with the same label increase.
In CKA and RSA, thesimilarity between two RSMs, K m 1 f 1 and K m 2 f 2 , defines the similarity between two NNs, as shown in Eq.2.
However, both K m 1 f 1 and K m 2 f 2 are affected by the same factor (i.e., a confounder): the similaritystructure of input dataset K 0 , which can cause spuriously high similarity.
Intuitively, (dis)similardata points (green stars and red stars in Figure 1) in the input space are likely to be (dis)similar in therepresentation space of the first few layers, regardless of how the NN operates, and the representationsimilarity structure of different NNs would be similar even for random NNs that have totally differentfunctional behaviors.
Hence, CKA/RSA naturally depends on the specifics of the dataset, and fordifferent datasets (e.g., from different domains) with different K 0 , comparing CKA/RSA across thedatasets may lead to inconsistent comparison results. This is undesirable especially when the goalof calculating the similarity measures for NNs is to quantify how similar the networks are in thefunctional level, which should not be affected by the specifics of the dataset at hand. We fix this by regressing out the input similarity K 0 from K mf 1 and K mf 2 , i.e., deconfounding.
Afterdeconfounding, if two data items are very similar in the NN latent space, it is because the NN madethem so and not because they were similar in the first place. In this way, the deconfounded metricmore directly focuses the comparison on the functional form of the NNs and is less affected by thestructure of the given dataset in the input space.
Figure 1 illustrates the spurious similarity using the CKA similarity measure as an example, andcompares that with the deconfounded dCKA (defined in the next section) on the first-layer of ResNets[31] with 20 random samples from CIFAR-10 [32] test set.
We consider two pairs of ResNets: 1.two random ResNets generated by adding different Gaussian noise, N (0 , 1) , to each parameter ofthe pretrained ResNet-18 2 on ImageNet [33]; 2.
the pretrained (PT) ResNet-18 and a finetuned (FT)
ResNet-18 on CIFAR-10.
We notice that CKA on random NNs is almost 1, and counterintuitivelyit is even higher than the CKA between PT and FT ResNets on a similar domain (0.99 vs. 0.95),although
we would expect the PT and FT networks to learn similar low-level features and hence bemore similar than random networks.
This happens because the similarities between samples in theinput space confound their similarities in the representation space.
After adjusting for the confounderwith dCKA, the similarity between the two random ResNets is much smaller than the similarity ofthe PT and FT networks (0.43 vs. 0.72).
Methods
In this section, we first propose a general fix on the NNs similarities (defined in Eq.1-2) by regressingout the confounder, i.e., the input similarity, in Section 3.1, and we give two examples of deconfoundedsimilarities: deconfounded CKA and deconfounded RSA in Section 3.2.
We then study the invarianceproperties of deconfounded similarities in Section 3.3.
3.1 Deconfounding representation similarity
We propose a simple approach to adjust the spurious similarity caused by the confounder by regressingout the input similarity structure from the representation similarity structure [34].
That is:
[Equation]
where K 0 is the input similarity structure, and ˆ α m 1 f
and ˆ α m 2 f 2 are the regression coefficients thatminimize the Frobenius norm of dK m 1 f 1 and dK m 2 f 2 respectively.
Furthermore, the letter d in front ofa similarity matrix, e.g. as in dK m 1 f 1 , denotes the deconfounded version of K m 1 f 1 , and similarly theletter d is applied throughout the text to denote all defounded quantities.
To do the deconfounding,we assume that the input similarity structure K 0 has a linear and additive effect on K mf , i.e.,
[Equation]
where vec ( · ) flattens a matrix to a vector. Noise ϵ mf is assumed to be independent from the confounderwith ˆ ϵ mf = vec ( dK mf ) , and
[Equation]
In Appendix A, we further verify that the linearity assumption in Eq.4 holds in general, i.e., addingadditional nonlinear terms of vec ( K 0 ) to Eq.4 does not improve the fit, and the noise term ϵ mf is notauto-correlated (solution Eq.5 is not misspecified) under the the Durbin-Watson test [35].
After the deconfounded similarity structures are obtained with Eq.3, we use the same similaritymeasure to calculate the deconfounded representation similarity:
[Equation]
Note that dK mf is not always positive semi-definite, even when K mf is positive semi-definite (PSD).
For a similarity measure s ( · , · ) that takes two kernel matrices as input, such as the CKA, we transformdK mf into a positive semi-definite matrix by removing all the negative eigenvalues according to [36].
Specifically, we have the eigenvalue decomposition of dK mf , such that
[Equation]
where λ i is the i th eigenvalue of dK mf .
We approximate
dK mf with a PSD matrix
[Equation] [Equation]
3.2 Examples of deconfounded similarity indices
Deconfounded CKA.
In CKA [19], the similarity structure in the feature space is represented with avalid kernel l ( · , · ) , i.e., K m 1 f 1 = l
( X m 1 f 1 , X m 1 f 1 ) and K m 2 f 2 =
l ( X m 2 f 2 , X m 2 f 2 ) , such as the linear or RBFkernel.
Then an empirical estimator of HSIC [37] is used to align two kernels: [Equation]
[Equation]
where H is the centering matrix.
CKA is given by the normalized HSIC such that
[Equation]
To deconfound the representation similarity matrices K m 1 f
and K m 2 f 2 , we apply the same kernel tomeasure the inter-example similarity in the input space K
= l ( X, X ) , and adjust its confoundingeffect with Eq.3.
However, matrices dK m 1 f 1 and dK m 2 f 2 , obtained by regressing out one kernelmatrix from another kernel, are no longer kernels, and they are not applicable for computing HSIC.
Fortunately, with Eq.8, we can approximate the dK m 1 f 1 and dK m 2 f 2 with two valid kernels ˜ dK m 1 f 1 and˜ dK m 2 f 2 , which are then used to construct the deconfounded CKA (dCKA):
[Equation]
We use linear kernels here because Kornblith et al.
[19] report similar results with RBF kernels.
Deconfounded RSA.
Different from CKA, the similarity structure in RSA [8] is measured by thepairwise Euclidean distance between examples in the feature space.
Specifically, each element of
K m 1 f 1 and K m 2 f 2 is obtained by K m 1 f 1 ,ij =
∥ x m 1 f 1 ,i − x m 1 f 1 ,j ∥ 2 and K m 2 f 2 ,ij = ∥ x m 2 f 2 ,i − x m 2 f 2 ,j ∥ 2 , wherex m 1 f 1 ,i is the m 1 -layer representation of the i th example in NN f 1 .
Thus, the input similarity structure
K 0 is measured with the pairwise Euclidean distance in the input space.
After K 0 is adjusted with
Eq.3, we apply Spearman’s ρ correlation to measure the similarity between the upper triangular partof dK m 1 f
and dK m 2 f 2 , i.e., triu ( dK m 1 f 1 ) and triu ( dK m 2 f 2 ) , that isdRSA
[Equation] [Equation] [Equation]
Note that rank correlation does not require two similarity matrices to be positive semi-definite.
Therefore, we skip the steps of constructing the PSD approximation.
Additional computational complexity.
The computational cost of the deconfounding (Eq.3) and
PSD approximation (Eq.8) steps are O ( n 2 ) and O ( n 3 ) respectively.
Therefore, dCKA has the samecomplexity as CKA: O ( n 3 ) .
For RSA, only the deconfounding step with O ( n 2 ) complexity is needed.
In experiments, computing dCKA between two XLM-RoBERTa models [38] takes 0 .
37 ± 0 .
11 slonger than CKA for each layer on 3000 random English sentences with a single 2080Ti GPU.
3.3 Theoretical properties
In this section, we study the invariance properties of the deconfounded representation similarity.
Forsimilarity measures that we studied, i.e., CKA and RSA, the corresponding deconfounded similarityindices have the same invariance properties, such as invariance to orthogonal transformation andisotropic scaling, as the original similarity measures.
Proposition 3.1.
Deconfounded CKA and deconfounded RSA are invariant to orthogonal transformation, if the (dis)similarity measure k ( · , · ) that compare inter-examples are orthogonal invariant.
Proposition 3.2.
Deconfounded CKA with a linear kernel and deconfounded RSA are invariant toisotropic scaling.
Intuitively, as long as k ( · , · ) is invariant to orthogonal transformation, e.g., linear kernels and Euclidean distance, the deconfounded representation similarity matrix dK mf in Eq.3 is also invariantto orthogonal transformation, because it is defined in terms of the kernel k .
Thus all operations ondK mf are invariant to orthogonal transformation.
Moreover, if one representation is scaled by a scalar,dK mf and ˜ dK mf will be scaled by the same scalar, whose effects will be finally eliminated in thenormalization step in CKA (Eq.10) and the rank correlation step in RSA (Eq.12). We give proofs in Appendix B.
Experiments
In this section, we design experiments to verify that deconfounding can improve the consistency of
CKA and RSA with the functional similarity of NNs from various perspectives.
First, in Section4.1, we check if these representation similarity measures can separate functionally similar NN pairsfrom random NN pairs.
In Section 4.2 we study how consistent the similarity measures are withsmall changes of NNs across different domains.
We then extend a recent framework [15], which usesrank correlation to compare the representation similarity and the functional similarity of NNs, totransfer learning on multi-domain image and natural language datasets (Section 4.3), and to conductchallenging out-of-distribution generalization tests [15] (Section 4.4).
4.1 Ability of detecting
functionally similar NN pairs from random NN pairs
Setup: We check if similarity measures can identify functionally similar NN representations fromrandom NN representations.
For each model block of ResNets (containing 2-3 convolutional layers),we generate two distributions of similarities: the null distribution H 0 and the alternative distribution
H 1 .
The H 0 contains similarities between 50 pairs of random ResNets on CIFAR-10 test set.
Wegenerate random NNs by randomly initializing the weight of an untrained ResNet-18 from N (0 , 10) [Figure] with different random seeds.
We also consider permuting the weight matrix of the pretrained ResNet18 to generate random NNs, which preserves parameter distributions, in Appendix D.1.
Distribution
H 1 contains similarities between the pretrained ImageNet NN and each of the 50 ResNets trained on
CIFAR-10 from scratch with different random initializations, on the same CIFAR-10 test as H 0 .
Results: We expect the similarities in H 1 to be significantly larger than those in H 0 .
Intuitively,models trained on ImageNet and CIFAR-10 are functionally similar in shallow blocks becausetheir domains are similar; hence similar low-level features are expected [39], while sufficientlyrandomized untrained NNs are functionally different.
We compute the proportion (shown in Figure
Left ) of 50 ImageNet-CIFAR NN pairs in H 1 whose similarity is significantly larger than H 0 , i.e.,larger than the upper bound of its 95% CI.
In Figure 2 Left , we observe that in shallow blocks thedeconfounded similarity can detect a larger proportion of ImageNet-CIFAR pairs from random NNpairs than the original measures.
For example, in the third and fourth block, dCKA can still detectall functionally similar pairs whereas the original CKA fails to detect any from random pairs.
Byaveraging proportions of all blocks, deconfounding increases CKA from 0.24 to 0.5 and RSA from0.02 to 0.25.
In Figure
Right , we visualize the alternative distribution H 1 and the null distributions
H 0 of each similarity measure as histograms in the first block.
We observe that although CKA anddCKA can identify the same proportion of ImageNet-CIFAR pairs in block 1, the difference between
H 1 and H 0 is more significant with the deconfounded similarities.
Moreover, in deep layers, e.g.,after block 4, no method can identify ImageNet-CIFAR pairs.
We hypothesise that ImageNet and
CIFAR-10 contain different classes of images, thus their high-level representations are significantlydifferent.
We discuss this more in Appendix D.
4.2 Consistency of NN functional similarities across domains
Setup: Ideally, a functional similarity between NNs would not depend on the domain in which thenetworks are applied.
Here, we study this by constructing a set of 6 NNs, { f i | i ∈ { 1 , 2 , 3 , 4 , 5 , 6 }} ,by adding independent Gaussian noise i × N (0 , 0 . 1) to each parameter of the pretrained ResNet-18,f ∗ .
Hence, the similarity s ( f i , f ∗ ) should be higher than s ( f i +1 , f ∗ ) regardless of the input domainin which the the networks are applied to calculate representations.
Thus, we calculate the similaritys ( f i , f ∗ ) for every f i on each domain of the corrupted CIFAR-10-C dataset [40] that contains 19domains with different types of corruptions to the original CIFAR-10.
We compute the averagesimilarity µ f i ,f ∗ across the 19 domains and its standard error σ f i ,f ∗ .
We say f i is significantly moresimilar to f ∗ than to f i +1 across domains, if
[Equation]
We repeat the above experiments 20 times with different random seeds, i.e., generate 20 different setsof NNs, to measure the proportion of cases where f i is significantly more similar to f ∗ than f i +1 foreach block, as well as the confidence interval of the proportion.
Results:
In Figure 3(A) Left , we show the proportion of identified NNs averaged over all blocksfor each noise level.
We observe that the deconfounded similarity improves the proportion ofidentified NNs compared with the corresponding original similarity for all noise levels.
The averagedproportion increases 59 .
7% for CKA (from 0.18 to 0.29) and 43 .
1% for RSA (from 0.16 to 0.23)
[-We visualize the histogram of CKA and dCKA between f 2 and f ∗ and between f 
-]
0.1 0.2 0.3 0.4 0.5 noise level 0.
P r o p o r t i o n
CKA: 0.18 dCKA: 0.29 RSA: 0.
dRSA: 0.
8 block 0.
P r o p o r t i o n (A) 0.
d e n s i t y
CKA (overlap: 28.9%) 0.
dCKA (overlap: 13.5%)
Histogram, =0.2 Histogram, =0.
KDE, =0.2 KDE, =0.
(B) after deconfounding.
We also observe that deconfounding can improve CKA/RSA on different inputsfrom the same domain, but the improvement is relatively smaller ( 23% for CKA, from 0.65 to 0.8,and 7% for RSA, from 0.75 to 0.8), shown in Appendix E. Moreover, the proportion decreases asthe noise level increases for all similarity measures, because for large noise level (large i ), both f i +1and f i are far from f ∗ .
In Figure 3(A)
Right , we show the the proportion of consistently identifiedsimilarities for each block where the results are averaged over different noise levels.
In general, weexpect to identify fewer similar NNs with deeper layer representations, because the Gaussian noiseis added to each parameter and deeper representations consequently accumulate more noise thanshallow layers.
We visualize the histogram of CKA and dCKA between f 2 and f ∗ and between f 3and f ∗ on the first-block representations of inputs from 19 domains in Figure 3(B), where we canclearly observe that f 2 and f 3 are more separable in terms of dCKA than CKA.
Moreover, the relativeoverlap areas between two histograms are 13.5% and 28.9% for dCKA and CKA respectively.
4.3 Transfer learning
: domain similarity vs. the similarity of pretrained and finetuned NNs
Motivation: Ding et al.
[15] argued that the similarity metric must be sensitive to changes thataffect the functionality of the networks we compare, and we extend this to transfer learning underdomain shift.
Consider two models with the same initialization from a pretrained (PT) modelwhich are finetuned (FT) on data from different domains.
We then expect the similarity of the layerrepresentations between the finetuned and pretrained models to be different for each target domain,and ideally it should be correlated with the similarity between the source and target domains.
Setup: To study this in detail for dCKA and CKA, we choose datasets from two modalities – imageand text – that display such domain shift.
For text, we use the Multi-lingual STS-B dataset [41] andchoose English, Spanish, Polish, and Russian languages as the target domains.
For images, we usethe Real, Clipart, Sketch, and Quickdraw as target domains from the DomainNet dataset [42].
First, we finetune separate models for each domain from both modalities.
For text, we initialize a PT
XLM-RoBERTa [38] model and for images, we pretrain a ResNet-50 model on ImageNet [33].
Wecompute the layer-wise CKA and dCKA between each FT model and the corresponding PT model onthe test set of the target domain [22].
We repeat the above procedure 10 times to construct error bars.
Then, we quantify the similarity between two domains according to [43].
Specifically, we build across-domain classifier and create a dataset using an equal number of samples from each domain.
Next, we train a weak discriminator to predict the appropriate true domain for each sample.
Forthe discriminator, we use EfficientNet-B0 [44] for images and Distil-RoBERTa [45] for text.
Thediscriminator is essentially a binary classifier and thus the test binary cross entropy (BCE) loss wouldbe an indicator of the similarity between the two domains (higher meaning more similar).
Finally, wemeasure the Spearman’s ρ and Kendall’s τ correlation [15] between the domain similarity (measuredby the BCE loss of cross-domain classifier) and the layer-wise CKA and dCKA between PT and FTmodels of each domain.
We average the rank correlations over all layers.
Results: Table 1 summarizes the averaged rank correlations of CKA and dCKA for each modality.
We can see that the dCKA is more correlated with the domain similarity as compared to CKA on bothmodalities.
Figure
Left shows the results for the test BCE loss when the cross-domain classifier wastrained to discriminate between the domains on STS-B: English-English, English-Spanish, English- [Table] average similarities across layers dCKA is better correlated with domain Polish, and English-Russian, and we see that the similarity of two English domains is higher than thesimilarity of English and Russian domains, as expected.
The layer-wise CKA and dCKA between thepretrained and finetuned XLM-RoBERTa model is shown Figure 4 ( Center and Right ), respectively.
The models finetuned on highly dissimilar domains (e.g., English-Russian) are expected to have lowerlayer-wise similarity with the pretrained model.
However, the CKA gives counter-intuitively highsimilarities between PT and FT for English-Russian, and negative rank correlations in Table 1.
Thisis not because FT is close to PT, but because the Russian FT dataset is used to calculate the CKA,which causes the representations of PT and FT to be similar on that dataset.
As expected, dCKA(Figure 4 Right ) captures the domain similarity precisely by removing the input structure.
We provideresults on DomainNet in Appendix F.
4.4 Model generalization: in-distribution similarity vs. out-of-distribution accuracy
Setup:
Here we use a challenge task used in previous work [15], which evaluates the sensitiveness ofsimilarities to changes that affect the NNs generalizations on out-of-distribution (OOD) data, andwe would expect similar NNs to have similar OOD accuracy.
We follow the same setup as [15]: 1.
We train 50 ResNet-18, f i , with different random initialization on CIFAR-10; 2.
Evaluate the OODaccuracy of each model on CIFAR-10-C [40], acc ( f i ) , and select the most accurate ResNet as thereference model, f ∗ ; 3.
Compute the similarity between each f i and f ∗ , s ( f i , f ∗ ) , of each block on
CIFAR-10 test set (in-distribution similarity), and compute the accuracy difference on CIFAR-10-Ctest set | acc ( f i ) − acc ( f ∗ )
| (OOD accuracy similarity); 4.
Measure the Kendall’s τ and Spearman’s ρbetween 1 − s ( f i , f ∗ ) and | acc ( f i ) − acc ( f ∗ ) | for each block.
A good similarity should have a highrank correlation, meaning that the similarity in the input space correlates with OOD accuracy.
Results: CIFAR-10-C dataset contains 19 different corruptions and 5 levels for each corruption.
Weaverage rank correlations over all blocks of ResNet-18 as [15] because the ranking between similaritymeasures were shown to be consistent across different layers/blocks.
We report the averaged rankcorrelation over all types of corruptions in Table 2.
We observe that dCKA is more correlatedwith OOD accuracy on all 5 levels of corruption, especially on levels 3-5, compared with CKA.
Moreover, we also notice marginal improvements of dCKA in terms of in distribution accuracy (IDaccuracy).
Figure 5(A) shows the improvement of dCKA vs. CKA for each corruption averagedover 5 corruption levels, and we see that deconfounded CKA improves the most in the ‘contrast’corruption ( 15% ) and the least in the ‘zoom blur’ corruption ( 1% ).
We visualize examples of originalin-distribution images together with the above two corruptions in Figure 5(B), and we observe that‘contrast’ is very different from the in-distribution ‘CIFAR’ images whereas ‘zoom blur’ is more [Figure] [Table] [Figure]
We observe that ‘contrast’ is more different from the uncorrupted dataset compared with ‘zoom blur’, in terms of mean pair-wise distance (MD) of images.
similar to the original images visually. Moreover, we compute the mean pairwise Euclidean distances(MD) between images for each type.
We find the MD of the original CIFAR test set (MD = 658 . 7 )is close to the ‘zoom blur’ (MD= 513 . 7 ), and very far from the ‘contrast’ corruption (MD = 34 . 7 ).
Hence, the benefit of dCKA vs. CKA appear greatest when the OOD domain is least similar to theoriginal domain, which aligns with the expectation that dCKA is less domain specific due to thecorrection for the input domain population structure.
Discussion
We investigated the confounding effect of the input similarity structure on commonly used similaritymeasures between NNs representations.
The confounder can lead to a high similarity even forcompletely random NNs and counter-intuitive conclusions when NNs trained in multiple domains areconsidered.
We proposed a simple deconfounding algorithm by regressing out the input similarityfrom the representation similarity.
The deconfounded similarity measures studied in this paper retainthe invariance properties of the original measures.
Moreover, deconfounding significantly improvesthe consistency of similarities w.r.t.
functional similarities of NNs, which is especially beneficial inunderstanding NNs when multiple domains are involved, such as studying the closeness between thefinetuned and pretrained models in transfer learning (Figure 4).
Although the deconfounded similaritydoes not improve models’ performance directly, some insights can potentially inspire the futuredevelopment of better machine learning models. For instance, in (low-resource) transfer learning,we could encourage the dCKA between the finetuned and pretrained models to be correlated withthe known domain similarity during training, as we observed a high correlation between domainsimilarity and PT-FT similarity in Section 4.3. We further clarify that the original CKA can provide a meaningful representation similarity inapplications with a single domain, such as in understanding NN trained with different initializations[19] or with different architectures [21] on the same dataset, and in these cases dCKA is expected toyield similar conclusions with CKA (Appendix G).
Moreover, in specific neuroscience applications[46], the input similarity structure can be a feature that should not be removed, and dCKA isnot expected to be effective.
However, in applications involving multiple domains, such as transfer learning, we showed that deconfounding yields more intuitive results, and we expect other applicationswith multiple domains or data sets, such as meta learning, to benefit from the insights in this paper.
There are still few limitations and open questions.
We assumed that the confounder is linearlyseparable from the representation similarity in Eq.4, and showed that adding higher-order polynomialterms cannot improve the model evidence in Appendix A.
However, it is still possible that the inputsimilarity structure is not entirely additively separable from representation similarity structures,especially for deeper layers, and this may explain the fact that deconfounding similarities aremore beneficial for shallow layers (e.g., Figure 2 Left and Figure 3).
One possible solution isregressing out the similarity structure in the previous layer instead of the input layer, which improvesdetecting similar networks from random networks for deep blocks (Appendix D.2).
However, thisdiscards information from all previous layers and eventually loses the ability of representing thesimilarity between functional behaviors.
We consider this as an open question to motivate progresson developing more functionally consistent similarity measures.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes] See Section 2.2 to 4.
(b) Did you describe the limitations of your work?
[Yes] See Section 5.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Section
3.3 and Appendix A.
(b) Did you include complete proofs of all theoretical results?
[Yes] See Appendix B.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] We provide the code in the supplemental material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes] See Appendix C.
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See Appendix C.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] We cite the original
papers.
(b) Did you mention the license of the assets?
[Yes] See Appendix H.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
We include the code in the supplemental material.
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[No] All assets are publicly available under licenses.
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]
A Statistical checks of the deconfounding assumption in Eq.4A.1 Linearity assumption of the input similarity structure
In Eq.4, we assume that the representation similarity structures linearly depend on the input similaritystructure.
To validate if the linear assumption is sufficient, we check if adding higher-order polynomialterms of input similarity to the regression model (Eq.4) can help explain the representation similaritystructure.
We show the effect of adding higher-order polynomial terms on a pretrained ResNet-18(contains 20 layers in total) with CIFAR-10 inputs in Figure 6.
We observe that neither the R 2 northe Bayesian information criterion (BIC) [47], approximating the model evidence, changes muchwhen adding higher-order terms.
Although R 2 can be marginally improved by increasing the order inthe deeper layers (e.g., layer 20), we only consider the linear model in this paper for simplicity.
A.2 Independence assumption of the noise term across the dataset
In Eq.4, we consider the collection of distance between each pair of examples is the dataset ofthe linear regression.
A potential issue is that the noise term ϵ mf can be correlated for differentpairs, while we assume independent noise when fitting model Eq.4.
Because the noise ϵ mf,ij of thedistance between example i and j and ϵ m f,ik of the distance between example i and k contain the sameinformation about example i , which might induce correlation between ϵ mf,ij and ϵ mf,ik .
Checking noisecorrelation is important to justify if the model solution Eq.5 is misspecified.
For each i , we apply the Durbin-Watson (DW) test on { ϵ mf,ij |∀ j } , and average the test statistics overthe index i .
DW is always in [0 , 4] , and if there is no evidence of noise correlation, the test statisticsequals to 2.
Otherwise, the closer to 0 the statistic, the more evidence for positive correlation, whilethe closer to 4 means a negative correlation.
We show the histogram of DW statistics as well as theaveraged DW for each layer in Figure 6, where no serious noise correlation is observed.
We empirically observe that correcting for the confounder with a linear model is sufficient, especially for shallow layers (demonstrated with ResNet-
CIFAR-10)
B Proofs of propositionsB.1 Proof of Proposition 3.
If the similarity measure k ( · , · ) is invariant to orthogonal transformation, the representational similarity matrices are invariant to orthogonal transformation too.
Thus, for any two full-rank orthonormalmatrices U and V , such that U T U
=
I and V T V
=
I , we have:
[Equation]
and the deconfounded RSMs are also invariant to orthogonal transformation:
[Equation]
Therefore the deconfounded similarity is invariant to orthogonal transformation:
[Equation]
which completes the proof.
Note that the PSD approximation will not affect the orthogonal invariance property, because deconfounded RSMs are invariant to orthogonal transformations.
B.2 Proof of Proposition 3.
We use dRSA as an example to show the proof.
For any γ, θ ∈ R , we have:
[Equation]
because of using the Euclidean distance.
Moreover, the deconfounded RSMs are scaled with λ 2 andθ 2 as well, because the regression coefficient, α in Eq.5, is scaled with the same value.
Therefore,the rank correlation between two scaled deconfounded RSMs does not change because the rank isinvariant to scaling all objects.
The PSD approximation will not affect the isotropic scaling invariance property too, because the PSDapproximation matrix will be scaled at the same level.
C Training details of NNs
C.1 ResNet training on CIFAR-10
We trained 50 ResNet-18 models without any pretrained model from different initializations on the
CIFAR-10 training set.
We use 200 epochs with batch size 128.
We train models with SGD: learningrate 0.1, momentum 0.9, and weight decay 5e-4.
We also use cosine annealing learning rate with
T max = 200 .
The averaged accuracy of trained models on CIFAR-10 test set is 0 . 89 , with standarddeviation 0 .
Model training takes around 10 hours.
C.2 ResNet training on DomainNet
We finetune Resnet-50 models which had been pretrained on the Imagenet dataset.
Separate modelsare trained on each domain of the DomainNet dataset, namely Real, Clipart, Sketch and Quickdraw.
The DomainNet task is to classify each image among 345 different classes.
For each domain, werepeat the training for 10 random restarts.
To keep the training uniform, we sample 5000 images fromeach domain and train the Resnet model for 2000 iterations with a batch size of 32.
All the inputimages are resized to 224 × 224 pixels, and we perform no other form of data augmentation.
Weuse the AdamW optimizer [48] with a base learning rate of 1 e − 3 , which was varied using a cosineannealing scheduler with a warm-up of 600 steps.
The fine-tuning takes around 8 hours in total forall random seeds.
Figure 7 Left shows the F1 scores of the finetuned model for each domain.
As expected, the scorefor the Real domain is the highest since it is the most similar to Imagenet.
The higher score on the
Quickdraw domain is possibly due to the simplicity of the Quickdraw input distribution (since itconsists of doodles) as compared to the other domains.
C.3 XLM-RoBERTa training on STS-B
Since we are training on different languages, we use the XLM-RoBERTa model as the base modeland finetune it on different STS-B tasks on English, Spanish, Polish and Russian languages.
Forfinetuning we use AdamW optimizer with a learning rate of 2 e − 5 which is linearly annealed, with30% of total steps for warmup, for 3 epochs.
We use a batch size of 8 and regularize the training witha weight decay of 0 . 01 .
The fine-tuning takes around 5 hours in total for all random seeds.
{+Figure 
+}
Right shows Spearman’s correlation of the predicted sentence similarity with the ground truth.
The performance on English is the highest while that on Russian is the lowest.
This performanceacross the languages correlates perfectly with the domain similarity reported in Section 4.3.
C.4 Computational resources
We used two GPUs: 2080Ti and Tesla V100, in our internal cluster.
D Additional experiments on detecting similar NNs from random NNs.D.1 Generate random NNs with permutation
Different from the main text where we consider random NNs as untrained NNs with differentinitializations, here we permute the weight matrix of each layer given a pretrained ResNet-18 on
ImageNet.
This can preserve the parameter distributions of the pretrained ResNet while generatingrandom functions.
From Figure 8, we observe that the conclusion drawn in the main text also holdswhen the null distribution is constructed with permutation.
[Figure]
D.2 Recursive deconfounding
From Table 2 Left , we observe that although deconfounded similarities improve detecting semanticallysimilar NNs from random NNs, no similarity can identify ImageNet-CIFAR pairs for deep layers.
One hypothesis is that the confounding effect of input similarity cannot be approximated well withadditively separable functions for deeper layers, because of the model nonlinearity added by each
NN layer.
Here we consider a natural extension of deconfounding input similarity: instead of regressing outthe input similarity directly (in Eq.3), we regress out the representation similarity structure from theprevious layer recursively:
[Equation]
Although Eq.18 has the same additively separable assumption as Eq.3, but Eq.18 is easier to satisfybecause it only assumes additively separable for one layer.
We call the similarity generated by Eq.18recursive deconfounded similarity.
We apply recursive deconfounded similarity, such as rdCKA and rdRSA, on the experiments ofdetecting similar networks from random networks described in Section 4.1.
We show the comparison results between dCKA/dRSA with rdCKA/rdRSA in Table 3, where we observe a marginal
[Table]
improvement from dCKA to rdCKA but a significant improvement from dRSA to rdRSA.
However,the proportion of identified similar networks is still low for deeper layers.
Thus, we consider that
ImageNet and CIFAR-10 learn different high-level representations because they contain differentclasses of images, as mentioned in the main text.
We visualize the histogram of each similarity measure for the last 3 blocks in Figure 9.
We observethat CKA/dCKA and RSA/dRSA are much smaller than two null distributions, while rdCKA/rdRSAcan have a similar level as the corresponding null distribution.
E Consistency of in-domain NN
functional similarities.
In Section 4.2, we test the consistency of different NN similarities across 19 different domains.
Inthis section, we verify if NN similarities are consistent across different input samples from the samedomain.
We repeat the same procedure described in Section 4.2, except that we calculate the similaritys ( f i , f ∗ ) on 20 different sets of examples sampled from the same domain, i.e., the CIFAR-10 test set,instead of 19 different OOD domains.
We observe that the deconfounded similarity can still identify more similar models compared with the original similarity
We show the results in Figure 10, where we observe that deconfounding can improve CKA/RSA withdifferent inputs from the same domain, but the improvement is marginal compared with cross-domainexperiments: 23% for CKA (from 0.65 to 0.8) and 7% for RSA (from 0.75 to 0.8), although theproportion of identified similar NNs are much larger than the cross-domain examples for all similaritymeasures.
F Layer-wise CKA and dCKA on DomainNet
We show the results of the DomainNet transfer learning tasks (Section 4.3) in Figure 11.
We observethat dCKA gives more consistent ranking w.r.t.
the domain similarity in middle blocks.
[Figure]
G Consistency of dCKA for NNs trained with different initialization
One advantage of original CKA is that it can reveal consistent relationships between layers of NNstrained with different initializations, while other representation similarity measures, such as PWCCAand Procrustes, can not [19, 15].
Here we show that dCKA has a similar behavior as CKA whenstudying the similarity between representations of NNs with different initializations given one domain.
As in [15], we first take 5 ResNet-18 models trained on CIFAR-10 dataset with different initializations.
For one model (model i ), we compute the similarity between every pair of layers of the model.
Wethen choose another model with a different initialization (model j ), and compute the similaritybetween every layer of model i and every layer of model j .
We average the results for five differentmodels.
We show the results in Figure 12, where we observe that for both CKA and dCKA and foreach layer of the model i , the most similar layer in model j is the same corresponding layer.
Hence,similarly to CKA, dCKA can identify consistent relationships of layers between different networks.
H Licenses
The source code and pretrained model for XLM-RoBERTa are available at https://huggingface.co/transformers/v4.9.2/model_doc/xlmroberta.html under the Apache License 2.0.
The source code of
Distil-RoBERTa are available at https://huggingface.co/distilroberta-base under the Apache License 2.0.
The source code and pretrained model of ResNets on ImageNet are available at https://pytorch.org/hub/pytorch_vision_resnet/ under license https://github.com/pytorch/pytorch/blob/master/LICENSE .
The source code of EfficientNet-50 are available at https://pytorch.org/hub/nvidia_deeplearningexamples_efficientnet/ under license https://github.com/pytorch/pytorch/blob/master/LICENSE
The DomainNet dataset is available from http://ai.bu.edu/M3SDA/#dataset with copyright information on the same website
The STS
B dataset is available from https
//paperswithcode.
com
dataset
sts-
benchmarkunder
CC-BY-SA
[Figure]
L a y e r
CKA
[Figure]
[Figure]
Layer     dCKA 0.
Model i
Model j
Model i
Model j
M ode l i (A) (B)
The CIFAR-10 dataset is available from https://www.cs.toronto.edu/~kriz/cifar.htmlunder MIT license.
The CIFAR-10-C dataset is available from https://zenodo.org/record/2535967#.YoThrKjP2Uk under CC-BY 4.0.