Deconfounded Representation Similarity for Comparison of Neural Networks
Anonymous Author(s)
AffiliationAddress email
Abstract
Similarity metrics such as representational similarity analysis (RSA) and centeredkernel alignment (CKA) have been used to understand neural networks by comparing their layer-wise representations.
However, these metrics are confounded by thepopulation structure of data items in the input space, leading to inconsistent conclusions about the functional similarity between neural networks, such as spuriouslyhigh similarity of completely random neural networks and inconsistent domainrelations in transfer learning.
We introduce a simple and generally applicablefix to adjust for the confounder with covariate adjustment regression, which improves the ability of CKA and RSA to reveal functional similarity and also retainsthe intuitive invariance properties of the original similarity measures.
We showthat deconfounding the similarity metrics increases the resolution of detectingfunctionally similar neural networks across domains.
Moreover, in real-worldapplications, deconfounding improves the consistency between CKA and domainsimilarity in transfer learning, and increases correlation between CKA and modelout-of-distribution accuracy similarity.
Introduction
Deep neural networks (NNs) have achieved state-of-the-art performance on a wide range of machinelearning tasks by automatically learning feature representations from data [1, 2, 3, 4, 5].
However,these networks do not offer interpretable predictions on most applications and are seen as “blackboxes”.
It is thus crucial to understand the intricacies of neural networks before they are deployedon critical applications.
Previous work has made progress in understanding how a single neuralnetwork makes decisions with axiomatic attribution methods [6, 7] and understanding how multipleneural networks relate to each other with representation similarity measures [8].
Several similaritymeasures between representations have been proposed with different principles, including linearregression [9], canonical correlation analysis (CCA; [10, 11]), statistical shape analysis [12] andfunctional behaviors on down-stream tasks [13, 14, 15].
Another main-stream approach is based onrepresentational similarity analysis (RSA, [8, 16, 17, 18]) and centered kernel alignment (CKA, [19]),which compute the similarity between (dis)similarity matrices of two neural network representationson the same dataset.
RSA and CKA have been successfully applied to understand biological [20] and artificial NNs [21] bystudying the similarity of representations of different NNs in a single data domain.
However, we findthat the inter-example (dis)similarity matrices in the representation space of different NNs are highlycorrelated with a shared factor (i.e., a confounder): the (dis)similarity structure of the data items inthe input space, especially for shallow layers.
This confounding issue limits the ability of CKA toreveal similarity of models on the functional level [15], and can result in a spuriously high CKA evenbetween two random NNs.
Moreover, this can lead to counter-intuitive conclusions when using CKA deconfounding
[Figure]
dCKA
[Figure]
CKA and dCKA between two random ResNets ( NN 1, NN 2 ) deconfounding
CKA and dCKA between pretrained ( PT ) and fine-tuned ( FT ) ResNets
CKA=0.
dCKA=0.
[Figure]
to compare models trained in multiple domains and/or to deduce functional similarities of NNs, suchas in understanding transfer learning, multitask learning, and meta learning [22, 23, 24, 25, 26].
In this paper, we propose to adjust the representation similarity to improve its consistency w.r.t.
thefunctional similarity by regressing out the confounder, the inter-example (dis)similarity matrix inthe input space, from the (dis)similarity matrices of two representations, inspired by the covariateadjusted correlation analysis widely studied in biostatistics [27, 28].
This approach is simple andwidely applicable on any similarity measure built on the CKA and RSA framework.
Moreover, westudy the invariance properties of the deconfounded representation similarity and demonstrate itsbenefits on public image and natural language datasets with various NN architectures.
Overall, our contributions are:
• We study the confounding effect of the input inter-example similarity on the representationsimilarity between two NNs, which limits CKA and RSA from revealing consistent func-
tional similarities.
We propose a simple and generally applicable deconfounding fix anddiscuss the invariance properties of the deconfounded similarities.
• We verify that deconfounded similarities can detect functionally similar NNs from randomNNs, and small NN changes across domains where previous similarity measures fail.• We show that deconfounded similarities are more consistent with domain similarities in
transfer learning, on both image and language datasets, compared with existing methods.•
We demonstrate that deconfounded similarities on in-distribution datasets are more corre-
lated with out-of-distribution accuracy than the corresponding original similarities [15].
Preliminaries
2.1 Notation and prior work
Let X ∈ R n × p denote the input dataset with n datapoints and p features, and X m 1 f 1 ∈ R n × p 1 and
X m 2 f 2 ∈ R n × p 2 denote the m 1 th and m 2 th layer representations of two NNs of interest, f 1 ( X ) andf 2 ( X ) , with p 1 and p 2 nodes respectively.
We center and normalize the representation matrices byfirst removing the mean of each feature (i.e., each column), and then dividing by the Frobenius norm.
A standard approach for comparing representations of two NNs is to compare the similarity structuresin each network representation.
This can be done by first computing the similarity between every pairof examples in X m 1 f 1 and X m 2 f 2 with a similarity measure k ( · , · ) :
[Equation]
Here K m 1 f 1 , K m 2 f 2 ∈ R n × n are called representational similarity matrices (RSMs) 1 .
Second, anothersimilarity measure s ( · , · ) is applied to compare these two similarity structures, K m
[Equation]
and K m
[Equation] [Equation]
This gives the similarity between the two representations.
The existing approaches vary by using different similarity measures for both levels of comparison.
CKA [19] employs a kernel function for the first level similarity k ( · , · ) , and a Hilbert-Schmidt
Independence Criterion (HSIC) estimator for the second, s ( · , · ) .
On the other hand, RSA-basedmethods use Euclidean distance to measure the inter-example dis similarity structure, and apply
Pearson’s correlation [8] or Spearman’s rank correlation [18] to quantify the similarity between twodissimilarity structures.
Although other approaches, e.g., linear regression [29] and CCA [10], havebeen proposed to compare NN representations, we focus on CKA and RSA in this paper due to theirwide usage in understanding the properties of NNs, such as transfer learning [22, 23, 30].
2.2 Illustration of the confounding in representation similarity
As shown in Eq.2, the similarity between two RSMs, K m 1 f 1 and K m 2 f 2 , defines the similarity betweentwo NNs.
However, both these similarity structures are affected by the same factor (i.e., a confounder):the similarity structure of input dataset X , which can cause spuriously high similarity.
Intuitively,(dis)similar data points (green stars and red stars in Figure 1) in the input space are likely to be(dis)similar in the representation space of the first few layers, regardless of how the NN operates.
Thusthe representation similarity structure of different NNs would be similar even for random NNs thathave totally different functional behaviors.
This is undesirable especially when the goal of calculatingthe similarity measures for NNs is to quantify how similar the networks are in the functional level,which should not be affected by the specifics of the dataset at hand.
Figure 1 illustrates this problem using the CKA similarity measure as an example, and compares thatwith the deconfounded dCKA (defined in the next section) on the first-layer of ResNets [31] with 20random samples from CIFAR-10 [32] test set. We consider two pairs of ResNets: 1. two random
ResNets generated by adding different Gaussian noise, N (0 , 1) , to each parameter of the pretrained ResNet-18 2 on ImageNet [33]; 2.
the pretrained (PT) ResNet-18 and a finetuned (FT) ResNet-18on CIFAR-10.
We notice that CKA on random NNs is almost 1, and counterintuitively it is evenhigher than the CKA between PT and FT ResNets on a similar domain (0.99 vs. 0.95), although wewould expect the PT and FT networks to learn similar low-level features and hence be more similarthan random networks.
This happens because the similarities between samples in the input spaceconfound their similarities in the representation space.
After adjusting for the confounder with dCKA,the similarity between the two random ResNets is much smaller than the similarity of the PT and FTnetworks (0.43 vs. 0.72).
Methods
In this section, we first propose a general fix on the NNs similarities (defined in Eq.1-2) by regressingout the confounder, i.e., the input similarity, in Section 3.1, and we give two examples of deconfoundedsimilarities: deconfounded CKA and deconfounded RSA in Section 3.2.
We then study the invarianceproperties of deconfounded similarities in Section 3.3.
3.1 Deconfounding representation similarity
We propose a simple approach to adjust the spurious similarity caused by the confounder by regressingout the input similarity structure from the representation similarity structure [34].
That is:
[Equation]
3) where K 0 is the input similarity structure, and ˆ
α m 1 f
and ˆ α m 2 f 2 are the regression coefficients thatminimize the Frobenius norm of dK m 1 f 1 and dK m 2 f 2 respectively.
Furthermore, the letter d in front of a similarity matrix, e.g. as in dK m 1 f 1 , denotes the deconfounded version of K m 1 f 1 , and similarly theletter d is applied throughout the text to denote all defounded quantities.
To do the deconfounding,we assume that the input similarity structure K 0 has a linear and additive effect on K mf , i.e.,
[Equation]
where vec ( · ) flattens a matrix to a vector.
Noise ϵ mf is assumed to be independent from the confounderwith ˆ ϵ mf = vec ( dK mf ) , and
[Equation]
In Appendix A, we further verify that the linearity assumption in Eq.4 holds in general, i.e., addingadditional nonlinear terms of vec ( K 0 ) to Eq.4 does not improve the fit, and the noise term ϵ mf is notauto-correlated (solution Eq.5 is not misspecified) under the the Durbin-Watson test [35].
After the deconfounded similarity structures are obtained with Eq.3, we use the same similaritymeasure to calculate the deconfounded representation similarity:
[Equation]
Note that dK mf is not always positive semi-definite, even when K mf is positive semi-definite (PSD).
For a similarity measure s ( · , · ) that takes two kernel matrices as input, such as the CKA, we transformdK mf into a positive semi-definite matrix by removing all the negative eigenvalues according to [36].
Specifically, we have the eigenvalue decomposition of dK mf , such that
[Equation]
where λ i is the i th eigenvalue of dK mf .
We approximate
dK mf with a PSD matrix ˜ dK mf :
[Equation]
3.2 Examples of deconfounded similarity indices
Deconfounded CKA.
In CKA [19], the similarity structure in the feature space is represented with avalid kernel l ( · , · ) , i.e., K m 1 f 1 = l
( X m 1 f 1 , X m 1 f 1 ) and K m 2 f 2 =
l ( X m 2 f 2 , X m 2 f 2 ) , such as the linear or RBFkernel.
Then an empirical estimator of HSIC [37] is used to align two kernels:
[Equation]
where H is the centering matrix.
CKA is given by the normalized HSIC such that
[Equation]
To deconfound the representation similarity matrices K m 1 f
and K m 2 f 2 , we apply the same kernel tomeasure the inter-example similarity in the input space K
= l ( X, X ) , and adjust its confoundingeffect with Eq.3.
However, matrices dK m 1 f 1 and dK m 2 f 2 , obtained by regressing out one kernelmatrix from another kernel, are no longer kernels, and they are not applicable for computing HSIC.
Fortunately, with Eq.8, we can approximate the dK m 1 f 1 and dK m 2 f 2 with two valid kernels ˜ dK m 1 f 1 and˜ dK m 2 f 2 , which are then used to construct the deconfounded CKA (dCKA):
[Equation]
We use linear kernels here because Kornblith et al.
[19] report similar results with RBF kernels.
Deconfounded RSA.
Different from CKA, the similarity structure in RSA [8] is measured by thepairwise Euclidean distance between examples in the feature space.
Specifically, each element of
K m 1 f 1 and K m 2 f 2 is obtained by K m 1 f 1 ,ij =
∥ x m 1 f 1 ,i − x m 1 f 1 ,j ∥ 2 and K m 2 f 2 ,ij = ∥ x m 2 f 2 ,i − x m 2 f 2 ,j ∥ 2 , wherex m 1 f 1 ,i is the m 1 -layer representation of the i th example in NN f 1 .
Thus, the input similarity structure
K 0 is measured with the pairwise Euclidean distance in the input space.
After K 0 is adjusted with
Eq.3, we apply Spearman’s ρ correlation to measure the similarity between the upper triangular partof dK m 1 f
and dK m 2 f 2 , i.e., triu ( dK m 1 f 1 ) and triu ( dK m 2 f 2 ) , that isdRSA ( K m 1 f 1 , K m 2 f 2 )
= ρ ( triu ( dK m 1 f 1 ) , triu ( dK m 2 f 2 )) .
Note that rank correlation does not require two similarity matrices to be positive semi-definite.
Therefore, we skip the steps of constructing the PSD approximation.
Additional computational complexity.
The computational cost of the deconfounding (Eq.3) and
PSD approximation (Eq.8) steps are O ( n 2 ) and O ( n 3 ) respectively.
Therefore, dCKA has the samecomplexity as CKA: O ( n 3 ) .
For RSA, only the deconfounding step with O ( n 2 ) complexity is needed.
In experiments, computing dCKA between two XLM-RoBERTa models [38] takes 0 .
37 ± 0 .
11 slonger than CKA for each layer on 3000 random English sentences with a single 2080Ti GPU.
3.3 Theoretical properties
In this section, we study the invariance properties of the deconfounded representation similarity.
Forsimilarity measures that we studied, i.e., CKA and RSA, the corresponding deconfounded similarityindices have the same invariance properties, such as invariance to orthogonal transformation andisotropic scaling, as the original similarity measures.
Proposition 3.1.
Deconfounded CKA and deconfounded RSA are invariant to orthogonal transformation, if the (dis)similarity measure k ( · , · ) that compare inter-examples are orthogonal invariant.
Proposition 3.2.
Deconfounded CKA with a linear kernel and deconfounded RSA are invariant toisotropic scaling.
Intuitively, as long as k ( · , · ) is invariant to orthogonal transformation, e.g., linear kernels and Euclidean distance, the deconfounded representation similarity matrix [Equation] in Eq.3 is also invariantto orthogonal transformation, because it is defined in terms of the kernel k .
Thus all operations ondK mf are invariant to orthogonal transformation.
Moreover, if one representation is scaled by a scalar,dK mf and ˜ dK mf will be scaled by the same scalar, whose effects will be finally eliminated in thenormalization step in CKA (Eq.10) and the rank correlation step in RSA (Eq.12).
We give proofs in
Appendix B.
Experiments
In this section, we design experiments to verify that deconfounding can improve the consistency of
CKA and RSA with the functional similarity of NNs from various perspectives.
First, in Section4.1, we check if these representation similarity measures can separate functionally similar NN pairsfrom random NN pairs.
In Section 4.2 we study how consistent the similarity measures are withsmall changes of NNs across different domains.
We then extend a recent framework [15], which usesrank correlation to compare the representation similarity and the functional similarity of NNs, totransfer learning on multi-domain image and natural language datasets (Section 4.3), and to conductchallenging out-of-distribution generalization tests [15] (Section 4.4).
4.1 Ability of detecting functionally similar NN pairs from random NN pairs
Setup: We check if similarity measures can identify functionally similar NN representations fromrandom NN representations.
For each model block of ResNets (containing 2-3 convolutional layers),we generate two distributions of similarities: the null distribution H 0 and the alternative distribution
H 1 .
The H 0 contains similarities between 50 pairs of random ResNets on CIFAR-10 test set.
Wegenerate random NNs by randomly initializing the weight of an untrained ResNet-18 from N (0 , 10)with different random seeds.
We also consider permuting the weight matrix of the pretrained ResNet18 to generate random NNs, which preserves parameter distributions, in Appendix D.1.
Distribution
H 1 contains similarities between the pretrained ImageNet NN and each of the 50 ResNets trained on
CIFAR-10 from scratch with different random initializations, on the same CIFAR-10 test as H 0 .
Results: We expect the similarities in H 1 to be significantly larger than those in H 0 .
Intuitively,models trained on ImageNet and CIFAR-10 are functionally similar in shallow blocks becausetheir domains are similar; hence similar low-level features are expected [39], while sufficiently 4 Blocks 0. P r o p o r t i o n CKA   4 Blocks
RSA CKA (0.24) dCKA (0.5)
[-We compute the proportion (shown in Figure 
-]RSA (0.02) dRSA (0.25) 0. B l o c k CKA 0. dCKA 0.85 0.
RSA 0.
dRSA similarity
{+[Figure] ImageNet-CIFAR NNs randomized untrained NNs are functionally different. We compute the proportion (shown in Figure 
+} Left ) of 50 ImageNet-CIFAR NN pairs in H 1 whose similarity is significantly larger than H 0 , i.e.,larger than the upper bound of its 95% CI.
In Figure 2 Left , we observe that in shallow blocks thedeconfounded similarity can detect a larger proportion of ImageNet-CIFAR pairs from random NNpairs than the original measures.
For example, in the third and fourth block, dCKA can still detectall functionally similar pairs whereas the original CKA fails to detect any from random pairs.
Byaveraging proportions of all blocks, deconfounding increases CKA from 0.24 to 0.5 and RSA from0.02 to 0.25.
In Figure
Right , we visualize the alternative distribution H 1 and the null distributions
H 0 of each similarity measure as histograms in the first block.
We observe that although CKA anddCKA can identify the same proportion of ImageNet-CIFAR pairs in block 1, the difference between
H 1 and H 0 is more significant with the deconfounded similarities.
Moreover, in deep layers, e.g.,after block 4, no method can identify ImageNet-CIFAR pairs.
We hypothesise that ImageNet and
CIFAR-10 contain different classes of images, thus their high-level representations are significantlydifferent.
We discuss this more in Appendix D.
4.2 Consistency of NN functional similarities across domains
Setup: Ideally, a functional similarity between NNs would not depend on the domain in which thenetworks are applied.
Here, we study this by constructing a set of 6 NNs, { f i | i ∈ { 1 , 2 , 3 , 4 , 5 , 6 }} ,by adding independent Gaussian noise i × N (0 , 0 . 1) to each parameter of the pretrained ResNet-18,f ∗ .
Hence, the similarity s ( f i , f ∗ ) should be higher than s ( f i +1 , f ∗ ) regardless of the input domainin which the the networks are applied to calculate representations.
Thus, we calculate the similaritys ( f i , f ∗ ) for every f i on each domain of the corrupted CIFAR-10-C dataset [40] that contains 19domains with different types of corruptions to the original CIFAR-10.
We compute the averagesimilarity µ f i ,f ∗ across the 19 domains and its standard error σ f i ,f ∗ .
We say f i is significantly moresimilar to f ∗ than to f i +1 across domains, if
[Equation]
We repeat the above experiments 20 times with different random seeds, i.e., generate 20 different setsof NNs, to measure the proportion of cases where f i is significantly more similar to f ∗ than f i +1 foreach block, as well as the confidence interval of the proportion.
Results:
In Figure 3(A) Left , we show the proportion of identified NNs averaged over all blocksfor each noise level.
We observe that the deconfounded similarity improves the proportion ofidentified NNs compared with the corresponding original similarity for all noise levels.
The averagedproportion increases 59 .
7% for CKA (from 0.18 to 0.29) and 43 .
1% for RSA (from 0.16 to 0.23)after deconfounding. We also observe that deconfounding can improve CKA/RSA on different inputsfrom the same domain, but the improvement is relatively smaller ( 23% for CKA, from 0.65 to 0.8,and 7% for RSA, from 0.75 to 0.8), shown in Appendix E. Moreover, the proportion decreases asthe noise level increases for all similarity measures, because for large noise level (large i ), both f i +1and f i are far from f ∗ . In Figure 3(A) Right , we show the the proportion of consistently identifiedsimilarities for each block where the results are averaged over different noise levels.
In general, weexpect to identify fewer similar NNs with deeper layer representations, because the Gaussian noiseis added to each parameter and deeper representations consequently accumulate more noise than
0.1 0.2 0.3 0.4 0.5 noise level 0.
P r o p o r t i o n
CKA: 0.18 dCKA: 0.29 RSA: 0.
dRSA: 0.
8 block 0.
P r o p o r t i o n (A) 0.
d e n s i t y
CKA (overlap: 28.9%) 0.
dCKA (overlap: 13.5%)
Histogram, =0.2 Histogram, =0.
KDE, =0.2 KDE, =0.
(B) shallow layers.
We visualize the histogram of CKA and dCKA between f 2 and f ∗ and between f 3and f ∗ on the first-block representations of inputs from 19 domains in Figure 3(B), where we canclearly observe that f 2 and f 3 are more separable in terms of dCKA than CKA.
Moreover, the relativeoverlap areas between two histograms are 13.5% and 28.9% for dCKA and CKA respectively.
4.3 Transfer learning: domain similarity vs. the similarity of pretrained and finetuned NNs
Motivation: Ding et al.
[15] argued that the similarity metric must be sensitive to changes thataffect the functionality of the networks we compare, and we extend this to transfer learning underdomain shift.
Consider two models with the same initialization from a pretrained (PT) modelwhich are finetuned (FT) on data from different domains.
We then expect the similarity of the layerrepresentations between the finetuned and pretrained models to be different for each target domain,and ideally it should be correlated with the similarity between the source and target domains.
Setup: To study this in detail for dCKA and CKA, we choose datasets from two modalities – imageand text – that display such domain shift.
For text, we use the Multi-lingual STS-B dataset [41] andchoose English, Spanish, Polish, and Russian languages as the target domains.
For images, we usethe Real, Clipart, Sketch, and Quickdraw as target domains from the DomainNet dataset [42].
First, we finetune separate models for each domain from both modalities.
For text, we initialize a PT
XLM-RoBERTa [38] model and for images, we pretrain a ResNet-50 model on ImageNet [33].
Wecompute the layer-wise CKA and dCKA between each FT model and the corresponding PT model onthe test set of the target domain [22].
We repeat the above procedure 10 times to construct error bars.
Then, we quantify the similarity between two domains according to [43].
Specifically, we build across-domain classifier and create a dataset using an equal number of samples from each domain.
Next, we train a weak discriminator to predict the appropriate true domain for each sample.
Forthe discriminator, we use EfficientNet-B0 [44] for images and Distil-RoBERTa [45] for text.
Thediscriminator is essentially a binary classifier and thus the test binary cross entropy (BCE) loss wouldbe an indicator of the similarity between the two domains (higher meaning more similar).
Finally, wemeasure the Spearman’s ρ and Kendall’s τ correlation [15] between the domain similarity (measuredby the BCE loss of cross-domain classifier) and the layer-wise CKA and dCKA between PT and FTmodels of each domain.
We average the rank correlations over all layers.
Results: Table 1 summarizes the averaged rank correlations of CKA and dCKA for each modality.
We can see that the dCKA is more correlated with the domain similarity as compared to CKA on bothmodalities.
Figure
Left shows the results for the test BCE loss when the cross-domain classifier wastrained to discriminate between the domains on STS-B: English-English, English-Spanish, EnglishPolish, and English-Russian, and we see that the similarity of two English domains is higher than thesimilarity of English and Russian domains, as expected.
The layer-wise CKA and dCKA between thepretrained and finetuned XLM-RoBERTa model is shown Figure 4 ( Center and Right ), respectively.
The models finetuned on highly dissimilar domains (e.g., English-Russian) are expected to have lowerlayer-wise similarity with the pretrained model.
However, the CKA gives counter-intuitively highsimilarities between PT and FT for English-Russian, and negative rank correlations in Table 1.
Thisis not because FT is close to PT, but because the Russian FT dataset is used to calculate the CKA,which causes the representations of PT and FT to be similar on that dataset.
As expected, dCKA [Table] (Figure 4 Right ) captures the domain similarity precisely by removing the input structure.
We provideresults on DomainNet in Appendix F.
4.4 Model generalization: in-distribution similarity vs. out
of-distribution accuracySetup
: Here we use a challenge task used in previous work [15], which evaluates the sensitiveness ofsimilarities to changes that affect the NNs generalizations on out-of-distribution (OOD) data, andwe would expect similar NNs to have similar OOD accuracy.
We follow the same setup as [15]: 1.
We train 50 ResNet-18, f i , with different random initialization on CIFAR-10; 2.
Evaluate the OODaccuracy of each model on CIFAR-10-C [40], acc ( f i ) , and select the most accurate ResNet as thereference model, f ∗ ; 3.
Compute the similarity between each f i and f ∗ , s ( f i , f ∗ ) , of each block on
CIFAR-10 test set (in-distribution similarity), and compute the accuracy difference on CIFAR-10-Ctest set | acc ( f i ) − acc ( f ∗ )
| (OOD accuracy similarity); 4.
Measure the Kendall’s τ and Spearman’s ρbetween 1 − s ( f i , f ∗ ) and | acc ( f i ) − acc ( f ∗ ) | for each block.
A good similarity should have a highrank correlation, meaning that the similarity in the input space correlates with OOD accuracy.
Results: CIFAR-10-C dataset contains 19 different corruptions and 5 levels for each corruption.
Weaverage rank correlations over all blocks of ResNet-18 as [15] because the ranking between similaritymeasures were shown to be consistent across different layers/blocks.
We report the averaged rankcorrelation over all types of corruptions in Table 2.
We observe that dCKA is more correlatedwith OOD accuracy on all 5 levels of corruption, especially on levels 3-5, compared with CKA.
Moreover, we also notice marginal improvements of dCKA in terms of in distribution accuracy (IDaccuracy).
Figure 5(A) shows the improvement of dCKA vs. CKA for each corruption averagedover 5 corruption levels, and we see that deconfounded CKA improves the most in the ‘contrast’corruption ( 15% ) and the least in the ‘zoom blur’ corruption ( 1% ).
We visualize examples of originalin-distribution images together with the above two corruptions in Figure 5(B), and we observe that‘contrast’ is very different from the in-distribution ‘CIFAR’ images whereas ‘zoom blur’ is moresimilar to the original images visually.
Moreover, we compute the mean pairwise Euclidean distances(MD) between images for each type.
We find the MD of the original CIFAR test set (MD = 658 . 7 )is close to the ‘zoom blur’ (MD= 513 . 7 ), and very far from the ‘contrast’ corruption (MD = 34 . 7 ).
Hence, the benefit of dCKA vs. CKA appear greatest when the OOD domain is least similar to theoriginal domain, which aligns with the expectation that dCKA less domain specific correction domain [Figure] [Table]
[Figure]
We observe that ‘contrast’ is more different from the uncorrupted dataset compared with ‘zoom blur’, in terms of mean pair-wise distance (MD) of images.
Discussion
We investigated the confounding effect of the input similarity structure on commonly used similaritymeasures between NNs representations.
The confounder can lead to a high similarity even forcompletely random NNs and counter-intuitive conclusions when NNs trained in multiple domains areconsidered.
We proposed a simple deconfounding algorithm by regressing out the input similarityfrom the representation similarity.
The deconfounded similarity measures studied in this paper retainthe invariance properties of the original measures.
Moreover, deconfounding significantly improvesthe consistency of similarities w.r.t.
functional similarities of NNs, which is especially beneficial inunderstanding NNs when multiple domains are involved, such as studying the closeness between thefinetuned and pretrained models in transfer learning (Figure 4).
We further clarify that the original CKA can provide a meaningful representation similarity inapplications with a single domain, such as in understanding NN trained with different initializations[19] or with different architectures [21] on the same dataset, and in these cases dCKA is expected toyield similar conclusions with CKA (Appendix G).
Moreover, in specific neuroscience applications[46], the input similarity structure can be a feature that should not be removed, and dCKA isnot expected to be effective.
However, in applications involving multiple domains, such as transferlearning, we showed that deconfounding yields more intuitive results, and we expect other applicationswith multiple domains or data sets, such as meta learning, to benefit from the insights in this paper.
There are still few limitations and open questions.
We assumed that the confounder is linearlyseparable from the representation similarity in Eq.4, and showed that adding higher-order polynomialterms cannot improve the model evidence in Appendix A.
However, it is still possible that the inputsimilarity structure is not entirely additively separable from representation similarity structures,especially for deeper layers, and this may explain the fact that deconfounding similarities aremore beneficial for shallow layers (e.g., Figure 2 Left and Figure 3).
One possible solution isregressing out the similarity structure in the previous layer instead of the input layer, which improvesdetecting similar networks from random networks for deep blocks (Appendix D.2).
However, thisdiscards information from all previous layers and eventually loses the ability of representing thesimilarity between functional behaviors.
We consider this as an open question to motivate progresson developing more functionally consistent similarity measures.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes] See Section 2.2 to 4.
(b) Did you describe the limitations of your work?
[Yes] See Section 5.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Section
3.3 and Appendix A.
(b) Did you include complete proofs of all theoretical results?
[Yes] See Appendix B.3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] We provide thecode in the supplemental material.
(b) Did you specify all the training details (e g., data splits, hyperparameters, how they were chosen)?
[Yes] See Appendix C.
(c) Did you report error bars (e g., with respect to the random seed after running experi-
ments multiple times)?
[Yes]
(d) Did you include the total amount of compute and the type of resources used (e g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See Appendix C.
If you are using existing assets (e g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] We cite the original
papers.
(b) Did you mention the license of the assets?
[Yes] See Appendix H.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
We include the code in the supplemental material.
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[No] All assets are publicly available under licenses.
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]
[-Figure 
-]