D OMINANT D ATAPOINTS AND THE B LOCK S TRUCTURE P HENOMENON IN N EURAL N ETWORK H IDDEN R EPRESENTATIONS
Anonymous authors
Paper under double-blind review
A BSTRACT
Recent work (Nguyen et al., 2021) has uncovered a striking phenomenon in largecapacity neural networks: they contain blocks of contiguous hidden layers with highly similar representations.
This block structure has two seemingly contradictory properties: on the one hand, its constituent layers have highly similar dominant ﬁrst principal components (PCs), but on the other hand, their representations, and their common ﬁrst PC, are highly dissimilar across different random seeds.
Our work seeks to reconcile these discrepant properties by investigating the origin of the block structure in relation to the data and training methods.
By analyzing properties of the dominant PCs, we ﬁnd that the block structure arises from dominant datapoints — a small group of examples that share similar image statistics (e.g. background color).
However, the set of dominant datapoints, and the precise shared image statistic, can vary across random seeds.
Thus, the block structure reﬂects meaningful dataset statistics, but is simultaneously unique to each model.
Through studying hidden layer activations and creating synthetic datapoints, we demonstrate that these simple image statistics dominate the representational geometry of the layers inside the block structure.
We also explore how the phenomenon evolves through training, ﬁnding that the block structure takes shape early in training, but the underlying representations and the corresponding dominant datapoints continue to change substantially.
Finally, we study the interplay between the block structure and different training mechanisms, introducing a targeted intervention to eliminate the block structure, as well as examining the effects of pretraining and Shake-Shake regularization.
I NTRODUCTION
Many modern successes of deep neural networks have used simple rules to systematically increase model capacity, often through scaling architecture depth and width (Tan & Le, 2019).
These large capacity models typically also maintain strong performance even in tasks with small amounts of training data.
This has led to their widespread use across different many applications, including data-scarce, high-stakes settings such as medical imaging (Wang et al., 2016; Liu et al., 2017).
However, recent work has shown that the representational structures of these large capacity models exhibit distinctive properties that are not present in shallower/narrower networks.
Speciﬁcally, when using linear centered kernel alignment (CKA) (Kornblith et al., 2019) to measure similarity between hidden representations of neural network layers, Nguyen et al.
ﬁnd that a large set of contiguous layers that share highly similar representations.
This is visible as a clear block structure in the heatmap of pairwise linear CKA similarities between layers (see Figure 1a).
This block structure phenomenon is robust, appearing both in models trained on natural image datasets and in models trained on medical imaging datasets (Figure 1b).
The emergence of the block structure as model capacity increases does not merely indicate a change in relationships among layer representations; it is also associated with changes in the properties of individual representations.
As shown in Figure 2a, layer representation inside the block structure have dominant ﬁrst principal components (PCs) that explain the majority of variance in activations, whereas the ﬁrst PCs of representations of layers outside the block structure, or in networks without block structure, explain far less variance. But counterintuitively, although layer representations and
[Figure] [Figure]
their dominant ﬁrst PCs are similar across the layers that make up the block structure, they are highly dissimilar across random seeds, as shown in Figure 2b.
Such representational inconsistencies have been linked to overﬁtting and poor generalization (Morcos et al., 2018).
The prevalence of the block structure, and this discrepancy in its properties, motivate a pressing fundamental question — is the block structure a sign of overﬁtting to idiosyncrasies of the data and training process, or does it pick up meaningful signals?
In this paper, we uncover answers to this question, investigating the origin of the block structure in relation to the data and training methods, and reconciling its contradictory behaviors.
Speciﬁcally, our ﬁndings are as follows:
• The dominant ﬁrst principal components of the layers that make up the block structure arise from a small number of dominant datapoints that share similar characteristics (e
g., background color).
But the set of dominant datapoints and their common characteristics can vary across training runs, leading to the observed block structure dissimilarities across random seeds.
• Like the block structure phenomenon, dominant datapoints can only be found in large capacity neural networks.
We show that if these dominant datapoints are excluded from the dataset used for representational analysis, the block structure effect disappears.
• We ﬁnd that dominant datapoints strongly activate the hidden layers corresponding to the block structure.
By constructing synthetic examples based off of their shared image characteristics, we show that these characteristics are indeed responsible for the high activation norms.
• The block structure emerges early in training, but early block structures have different representations and yield different dominant datapoints than the ﬁnal block structure at the end of training.
• We study whether it is possible to eliminate the block structure without interfering with generalization using a novel principal component regularization method.
We also show that alternative training mechanisms such as transfer learning and Shake-Shake regularization can reduce the block structure and yield more consistent representations across different training runs.
R ELATED W ORK
Previous work has studied certain propensities of deep neural networks in the standard training setting, such as their simplicity bias (Huh et al., 2021;
Valle-Perez et al., 2018; Nakkiran et al., 2019), texture bias (Baker et al., 2018; Geirhos et al., 2018; Hermann et al., 2019) and reliance on spurious correlations (McCoy et al., 2019; Geirhos et al., 2020; Ribeiro et al., 2016; Jo & Bengio, 2017; Hosseini & Poovendran, 2018).
Inspired by the ﬁndings of Nguyen et al. (2021) that deep and wide networks learn many layers with very similar representations, we seek to characterize the kind of signals these layers may overﬁt to.
Additionally, our work also explores how this signal varies over the course of training instead of focusing on a single, ﬁnal model.
To do so, we analyze the behavior of the internal representations of models of varied depths and widths, using methods for measuring similarity of neural network hidden representations (Kornblith et al., 2019;
Raghu et al., 2017; Morcos et al., 2018).
Besides shedding light on model training procedures (Gotmare et al., 2018; Neyshabur et al., 2020), features (Resnick et al., 2019; Thompson et al., 2019;
Hermann & Lampinen, 2020), and dynamics (Maheswaranathan et al., 2019), representational analysis has also furthered understanding of network internals in applications of deep learning, such as medicine (Raghu et al., 2019) and machine translation (Bau et al., 2019).
Our work is also related to previous attempts to understand the properties of overparameterized models (Zhang et al., 2016).
Most theoretical work in this area has focused on linear models, models with random features, or kernel settings (Belkin et al., 2018;
Hastie et al., 2019; Liang et al., 2020; Bartlett et al., 2020), all of which lack intermediate features, or involves networks without nonlinearities (Advani et al., 2020).
Our results suggest that the behavior of intermediate features of practical neural networks changes dramatically with increasing overparameterization, in ways that are not obvious from previous analysis.
E XPERIMENTAL S ETUP AND B ACKGROUND
Measuring Representation Similarity with CKA:
Centered kernel alignment (CKA) (Kornblith et al., 2019; Cortes et al., 2012) addresses several challenges in measuring similarity between neural network hidden representations including (i) their large size; (ii) neuronal alignment between different layers; and (iii) features being distributed across multiple neurons in a layer.
Like Nguyen et al.
(2021), we use the minibatch implementation of linear CKA with a batch size of 256 sampled without replacement from the test dataset, and accumulate statistics over 10 epochs to allow the minibatch estimator to converge.
More concretely, given k minibatches of n examples, and two layers having p 1 neurons and p 2 neurons each, minibatch CKA takes as inputs k pairs of centered activation matrices ( X 1 , Y 1 ) , . .
, ( X k , Y k ) where X i ∈ R n × p 1 and Y i ∈ R n × p 2 reﬂect the activations of these layers to the same minibatches.
It produces a scalar similarity score between 0 and 1 by averaging HSIC scores over minibatches:
[Equation]
where HSIC 1 is the unbiased estimator from Song et al.
The result is an estimator of CKA that converges to the same value regardless of the batch size.
We compute CKA between all layer representations, including before and after batch normalization, activations, and residual connections.
In experiments that involve tracking how a model’s internal properties (principal components of activations, representation similarity, etc.) change across epochs, we set batch normalization layers to be in training mode, to reduce the difﬁculty of adapting to batch statistics of the test set when the model has not converged.
The Block Structure Phenomenon: Nguyen et al.
(2021) use linear CKA to compute the representation similarity for all pairs of layers within the same model and visualizes the result as a heatmap (with x and y axes indexing the layers from input to output).
They ﬁnd a contiguous range of hidden layers with very high representation similarity (yellow squares on heatmaps in Figure 1a) in very deep or wide models, and call this phenomenon the block structure.
The block structure arises in networks that are large relative to the size of the training set — although small networks do not exhibit block structure when trained on CIFAR-10, they do exhibit block structure on smaller datasets.
Representations of the layers that comprise the block structure exhibit different representational geometry than other layers.
For layers inside the block structure, the ﬁrst principal component explains a large fraction of the variance in representations; this is not the case for the rest of the layers or for networks without the block structure (Nguyen et al., 2021).
We replicate this observation in
Figure 2a.
The substantial similarity between layers inside the block structure reﬂects a high degree of alignment of their ﬁrst principal components, as can be seen from the following decomposition of linear CKA for centered activation matrices X ∈ R n × p 1 , Y ∈ R n × p 2 :
[Equation]
where u iX ∈ R n and u iY ∈ R
n are the i th normalized principal components of X and Y , and λ iX and λ iY are the amounts of variance that these principal components explain (Kornblith et al., 2019).
As the fraction of variance explained by the ﬁrst principal component of each representation approaches 1, CKA becomes a measure of the squared cosine similarity between ﬁrst principal components (cid:104) u
1 X , u 1 Y (cid:105) 2 .
Nguyen et al.
(2021) conclude that the block structure preserves and propagates a dominant ﬁrst principal component across many hidden layers.
Datasets & Models: Our setup closely follows that of Nguyen et al.
and analyzes ResNets of varying depths and widths, trained on common image classiﬁcation datasets CIFAR-10 and CIFAR100 (Krizhevsky et al., 2009), as well as the medical imaging dataset Patch Camelyon (Veeling et al., 2018).
These datasets are chosen to reﬂect the image statistics found in different domains, and we observe that they all easily induce a block structure in reasonably sized ResNets.
The ResNet architecture design follows Zagoruyko & Komodakis (2016), with the layers distributed evenly between three stages — each marked by a different feature map size — and the number of channels is doubled after each stage.
To scale the model depth and width, we increase the number of layers and channels respectively.
In experiments involving Shake-Shake regularization (Gastaldi, 2017), the network is modiﬁed to have 3 branches that are combined in a stochastic fashion.
More information on hyperparameters can be found in Appendix A.
D OMINANT D ATAPOINTS AND H OW T HEY S HAPE THE B LOCK S TRUCTURE 4. D ATAPOINTS THAT A CTIVATE THE F IRST P RINCIPAL C OMPONENTS
Motivated by previous evidence that the block structure propagates a dominant principal component across its constituent layers, we attempt to interpret the signal carried by this PC by examining the distribution of values after projecting all test examples onto the ﬁrst PC of each layer’s activations.
We ﬁnd that the distribution is bimodal: while most values are concentrated within a certain range (in terms of magnitude), there exist some that are orders of magnitude bigger (Figure 3).
We call these datapoints with large projections dominant datapoints , and ﬁnd that they are consistent across the range of layers making up the block structure, as seen from each column of images in
Figure 3 showing the dominant datapoints for two different layers in the same block structure.
This explains why the ﬁrst principal components of different layer activations inside the block structure are highly similar (Figure 2a), an observation made earlier in (Nguyen et al., 2021).
Moreover, this dominant datapoint phenomenon is present only in networks that also exhibit a block structure.
As shown in Appendix Figure C.1, in networks without block structure, projections on the ﬁrst principal component are unimodally distributed and the corresponding datapoints differ between layers.
Dominant datapoints are visually similar.
In the left column of Figure 3, we observe that all datapoints have a blue background, although the precise shade of blue varies.
However, the background colors that the ﬁrst principal component picks up on depend on the random seed used to train the model.
The right column of Figure 3 shows the corresponding properties of an architecturally identical model trained from a different seed, where the dominant datapoints share white backgrounds instead.
Refer to Appendix C for visualizations of dominant images found in other tasks (CIFAR100, Patch Camelyon) and model architectures (wide ResNet).
We ﬁnd that besides background [Figure] [Figure] color, the dominant datapoints can also reﬂect other simple image patterns that are prevalent in the dataset, such as the appearance of large dark spots in histopathologic scans (Appendix Figure C.3).
Finally, the block structure observed in linear CKA heatmaps arises solely from dominant datapoints.
As seen in Figure 4, when the 10% most dominant datapoints are excluded from evaluation, the block structure is completely eliminated in all 3 training runs of ResNet-164 (1 × ) that we examined, and the fraction of variance explained by the ﬁrst PCs is substantially reduced.
For one seed, removing only the 1% most dominant datapoints is already enough to achieve this effect.
Thus, the block structure is completely determined by the dominant images, and is sensitive to the frequency of the dataset statistics that they capture. We investigate the effect of removing dominant datapoints on
CKA heatmaps computed with nonlinear kernels in Appendix Figure J.3.
[Figure]
4. D OMINANT D ATAPOINTS H AVE L ARGE A CTIVATION N ORMS
Having discovered dominant datapoints by projecting onto the ﬁrst principal component of the underlying representations, we next explore their implications on other properties of the internal representations, such as through layer activations.
We investigate what happens to the activations of a dominant example as it propagates through the network, and observe that it strongly activates the parts of the network with a visible block structure.
Figure 5 shows two dominant datapoints, for a
ResNet trained on CIFAR-10 that has a preference for white background images (top row), and for another ResNet trained on Patch Camelyon that responds to clear pink backgrounds (bottom row).
Both models contain the block structure in their internal representations, and we ﬁnd that in the corresponding layers, the activations of the dominant datapoints are substantially bigger than the median activations of the minibatches they are a part of.
4. C ASE S TUDY : I MAGE B ACKGROUNDS A S A D OMINANT D ATASET P ROPERTY
As background colors appear to be a common dataset characteristic that is picked up by many largecapacity models, we provide further evidence, through data and training manipulations, to conﬁrm that this is a real property of the hidden representations that emerges only with overparameterization.
First, we attempt to illustrate the connection between speciﬁc background colors, which vary across random seeds, and the layer activations.
To approximate this data statistic, we repeat only the top left pixel in each image across the entire dimensions of the image, obtaining solid color images.
These synthetic images indeed yield even larger activations compared to the dominant images they are taken from, and different initializations of architecturally identical networks respond to different synthetic images.
For instance, given the ResNet-164 (1 × ) model that has dominant datapoints containing a blue background (Figure 3), its hidden layers are further activated when all image pixels are replaced with the same shade of blue, but a solid white image produces considerably smaller activations (Figure 6).
In the same ﬁgure, we observe the opposite trend for another ResNet164 (1 × ) seed, which has been shown to pick up on white backgrounds (see Figure 3).
Refer to
Appendix E for similar analysis on the Patch Camelyon dataset.
Intervention: Color Augmentation:
Given earlier insights, a natural intervention to prevent the network from potentially picking up background color signal is adding color augmentation.
This includes randomly dropping color channels and jittering brightness, contrast, saturation, and hue of training images (Howard, 2013; Szegedy et al., 2015).
As shown in Figure 7, training with this data augmentation reduces the block structure effect in large capacity models as expected.
[Figure]
A c t i v a t i o n N o r m
[Figure] [Figure]
[Table]
E VOLUTION OF THE B LOCK S TRUCTURE DURING T RAINING
In the previous section, we characterize the signals the block structure propagates across its layers, and explore their implications on other aspects of the network internals.
Informed by these ﬁndings, we next explore what happens to the block structure and the dominant images over time, from initialization until the model converges, and how this process varies across different training runs.
Figure 8 shows the evolution of the internal representations of a ResNet-110 (1 × ) model as it is trained for 300 epochs on CIFAR-10, and tracks how similar each checkpoint is to the ﬁnal model.
We observe that some structure in the CKA heatmap is already present by the ﬁrst epoch, and the heatmap undergoes little qualitative change past epoch 20 (top set of plots).
However, when we inspect the corresponding dominant datapoints and compare the hidden representations between intermediate checkpoints and the ﬁnal model (bottom rows of plots), we ﬁnd that the block structure does not always carry the same information.
Instead, the representations, and corresponding groups of dominant datapoints, only stabilize much later in training.
We observe similar behavior for other models in Appendix F, and note that the differences in block structure representations across random seeds already take shape near the start of training as well.
Overall these ﬁndings suggest that the uniqueness of the block structure representations in large-capacity models can be attributed to both initialization parameters and the image minibatches the models receive throughout training.
To further explore the link between dominant datapoints and ﬂuctuations in network representations, in Appendix Figure F.4, we track the magnitude of the projected value on the ﬁrst PC of a single dominant datapoint found at the end of training, and ﬁnd that the value plummets at epochs when the internal representation structure diverges from that of the fully trained model (i.e., epochs 0, 120 and 240).
At these epochs, the dominant datapoint does not produce large activations either (bottom set
[Figure] of plots) This illustrates the signiﬁcance of the observed variations in dominant datapoints through the training of large-capacity models.
B LOCK S TRUCTURE AND T RAINING M ECHANISMS
Having observed how the internal representation structures — more speciﬁcally the dominant PCs of the layer representations — could vary signiﬁcantly over the course of training, we turn to examining the interplay between the block structure and the training mechanism.
Even though the block structure arises naturally with standard training, previous work has suggested that the block structure may be an indication of redundant modules in the corresponding networks (Nguyen et al., 2021).
Thus, it is natural to ask whether it is possible to train large-capacity models that do not have a block structure, and how such models perform compared to those with block structures.
Since the block structure reﬂects the similarity of a dominant ﬁrst principal component, propagated across a wide range of hidden layers (see Section 3), we study whether regularizing the ﬁrst principal components of layer activations would eliminate the block structure.
More speciﬁcally, we estimate the fraction of variance explained by the ﬁrst principal component of each layer using power iteration and penalize it in the loss when it exceeds 20%.
We provide full implementation details in
Appendix G.
The resulting heatmap, in Figure 9 top right, shows that not only does this eliminate the block structure from the internal representations, but surprisingly there is also no detrimental effect on performance (Appendix Table G.1).
We even observe small accuracy improvements on
CIFAR-100 and in the low-data regime, as shown in Appendix Table G.1.
Other standard training practices that are commonly used to boost performance are also effective at reducing or eliminating the block structure effect.
Shake-Shake regularization (Gastaldi, 2017) eliminates the block structure for all of the network sizes that we examine (Figure 9, bottom left), whereas transfer learning (Figure 9, bottom right) and training with smaller batch sizes (Appendix H) appear to reduce the appearance of the block structure, although blocks are still discernible in the largest models that we trained.
In addition to regularizing the block structure, these training methods also produce more similar representations across different training runs of the same architecture conﬁguration (Appendix Figure I.1).
Overall, our ﬁndings suggest that it is possible to obtain good generalization accuracy in networks with and without block structure.
We observe that learning processes that reduce the dominance of the ﬁrst PC of the representations provide slightly higher accuracies than standard training.
However, [Figure] some caution is warranted in interpreting these performance beneﬁts: it may be difﬁcult to causally determine the connection between the block structure and performance, as any training intervention targeting the block structure may have other distinct ramiﬁcations that also affect performance
[Figure]
D ISCUSSION
Scope and Limitations:
Our work primarily focuses on the behavior of large-capacity networks trained on relatively small datasets.
This is motivated by domains such as medical imaging where data is expensive relative to the cost of training a large model, and the high-stakes nature makes it important to understand the model’s behavior.
Additional future exploration is necessary to study state-of-the-art settings in e.g. NLP, which use much bigger and heterogeneous datasets.
Conclusion: The block structure phenomenon uncovered in previous work (Nguyen et al., 2021) reveals signiﬁcant differences in the representational structures of overparameterized neural networks and shallower/narrower ones.
However, it also exhibits some contradicting behaviors — being unique to each network while propagating a dominant PC across a wide range of layers — that suggest the underlying representations could either overﬁt to noise artifacts or capture relevant signals in the data.
Our work seeks to provide an explanation for this discrepancy.
We ﬁnd that despite the inconsistency of the block structure across different training runs, it arises not from noise, but real and simple dataset statistics such as background color.
We further discover a small set of dominant datapoints (with large activation norms) that are responsible for the block structure.
These datapoints emerge early in training and vary across epochs, as well as across random seeds.
We show how different training procedures, including color augmentation, transfer learning, ShakeShake regularization, and a novel principal component regularizer, can reduce the inﬂuence of these dominant datapoints, eliminating the block structure and leading to more consistent representations across training runs.
This work motivates interesting open questions such as exploring how dominant datapoints are manifested in other domains and applications of deep learning, and applying principal component regularization to distribution shift and self-supervision problems.
R EPRODUCIBILITY S TATEMENT
Our experiments are performed on standard datasets and architectures that are publicly available.
We provide information regarding training hyperparameters in Appendix A and implementation details for the principal component regularizer in Appendix G. We plan to release source code for reproducing our results upon acceptance of the paper.
Appendix A T RAINING D ETAILS
For wide ResNets, we look at models with depths of 14, 20, 26, and 38, and width multipliers of 1, 2, 4, 8 and 10.
For deep ResNets, we experiment with depths 32, 44, 56, 110 and 164.
In CIFARexperiments, the block structure only appears at a greater depth so we also include depths 218 and 224 in our investigation.
For Patch Camelyon datasets, we ﬁnd that depth 80 is enough to induce a block structure in the internal representations.
All ResNets follow the architecture design in (He et al., 2016; Zagoruyko & Komodakis, 2016).
Unless otherwise speciﬁed, we train all the models using SGD with momentum of 0.9 forepochs, together with a cosine decay learning rate schedule and batch size of 128.
Learning rate is tuned with values [0.005, 0.01, 0.001] and L 2 regularization strength with values [0.001, 0.005].
For
CIFAR-10 and CIFAR-100 experiments, we apply standard CIFAR-10 data augmentation consisting of random ﬂips and translations of up to 4 pixels.
With Patch Camelyon, we use random crops of size 32x32, together with random ﬂips, to obtain the training data.
At test time, the networks are evaluated on central crops of the original images.
For CKA analysis, each architecture is trained with 10 different seeds and evaluated on the full test set of the corresponding domain.
B B LOCK S TRUCTURE AND THE F IRST P RINCIPAL C OMPONENT
[Figure]
C A DDITIONAL V ISUALIZATIONS OF D OMINANT D ATAPOINTS
[Figure]
ResNet
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
1 × ) trained on CIFAR-
D B LOCK S TRUCTURE ON O UT - OF -D ISTRIBUTION D ATA
[Figure]
block structure depends on the data on which representations are computed.
We plot CKA heatmaps for models of varying depths (top rows) and widths (bottom rows) trained on CIFAR- 10, evaluated on different datasets ordered by the degree of out-of-distribution.
We observe that the block structure representation are robust to small distribution shifts in the data, as evident from CKAs computed on CIFAR-10 corrupted dataset (which adds perturbations to the original CIFAR-10 data) and CIFAR-100 dataset (which contains mutually exclusive classes but undergoes the same data collection procedure as CIFAR-10).
However, larger shifts, such as from CIFAR-10 to Patch Camelyon, produce signiﬁcantly different representational structures.
E D OMINANT E XAMPLES AND L AYER A CTIVATIONS
[Figure]
F E VOLUTION OF THE B LOCK S TRUCTURE
[Figure]
[Figure] [Figure]
[Figure] [Figure]
A c t i v a t i o n
N
[Figure]
r m
[Figure]
[Figure]
Similarity Between Layers of Same Checkpoint
[Figure]
(ResNet-164 (1 × )).
[Figure]
A c t i v a t i o n
[Figure] [Figure]
S t e p 0.
F r ac t
i o n V a r i a n ce E x p l a i n e d b y F i r s t
P C
G T RAINING WITH P RINCIPAL C OMPONENT R EGULARIZATION
To regularize the ﬁrst principal component, we ﬁrst compute the amount of variance that it explains using power iteration (Miyato et al., 2018).
At training step, t , we compute the batch of n convolutional feature maps with height h and width w containing c channels M t ∈ R n × h × w × c , ﬂatten the spatial dimensions to the channels dimension to create a matrix of size X t ∈ R n × p where p = h × w × c , and subtract its column means to obtain a centered matrix ˜ X t .
We randomly initialize the stored eigenvector u 0 ∈ R p at the beginning of training.
At each training step, we perform a single step of power iteration initialized from the previous eigenvector:
[Equation]
λ t approximates the top eigenvalue of ˜ X T t ˜ X t and thus the amount of variance explained by the ﬁrst principal component of the representation.
The proportion of variance explained is given by λ t / (cid:107) ˜
X t (cid:107) 2F .
We incorporate the regularizer as an additive term in the loss:
[Equation]
) where α is the strength of the regularizer and δ is the threshold proportion of variance explained at which it is imposed.
In our experiments, we tune α as a hyperparameter with values in [0.1, 1, 10], and set δ = 0 .
2 based on our analysis of the ﬁrst principal components of models without the block structure.
To speed up the training process, we only apply the regularizer to ReLU layers starting from the second stage, where block structure is often found.
[Table]
H E FFECT OF B ATCH S IZE ON THE B LOCK S TRUCTURE
[Figure]
I I MPACT OF T RANSFER L EARNING AND S HAKE -S HAKE R EGULARIZATION ON S IMILARITY OF L AYERS I NSIDE THE B LOCK S TRUCTURE
[Figure]
a
[Figure]
[Figure]
models with the same architecture but different initializations (off the diagonal) or within a single model (on the diagonal).
Similar to the obser- vation made in Figure I.1, while representations across models are highly dissimilar in the standard training case, models trained with principal component regularization from different random initializations show more representational similarity in corresponding layers.
J B LOCK S TRUCTURE U NDER D IFFERENT K ERNELS
As previously identiﬁed by (Nguyen et al., 2021), the block structure is a phenomenon the linear CKA heatmaps of large (wide or deep) networks. In this section, we investigate whether the block structure phenomenon also arises in CKA heatmaps computed with other kernels, and also examine the effect of removing the dominant datapoints (identiﬁed by the magnitudes of their projections on the ﬁrst principal component, as in Section 4.1) upon these CKA heatmaps. To compute CKA heatmaps under alternative kernels, we again use minibatch CKA.
The approach in Eq. 1 can be easily adapted to nonlinear kernels by replacing X i X T i and Y
i Y T i the linear Gram matrices formed by minibatch i , with minibatch kernel matrices K i ∈ R n × n and K (cid:48) i ∈ R n × n . The elements of these minibatch kernel matrices are the kernels between pairs of examples in the minibatches, i.e., K i lm = k ( X i l, : , X i m, : ) and K (cid:48) lm
= k (cid:48) ( Y i l, : , Y i m, : ) . Like linear minibatch CKA, nonlinear minibatch CKA is computed by averaging HSIC 1 across minibatches: [Equation]
[Equation]
We investigate the behavior of CKA under the linear kernel k linear ( x , y ) =
x T y , the cosine kernel k cos ( x , y ) =
x T y /
( (cid:107) x
cid:107)(cid:107) y (cid:107) ) , and the RBF kernel k rbf ( x , y ; σ ) =
exp( −(cid:107) x − y (cid:107) 2 / (2 σ 2 )) .
For each layer, we measure the median Euclidean distance ˜ d between examples in each layer and set σ = c ˜ d with c ∈ { 0 . 2 , 0 . 5 , 1 , 2 , 5 , 10 } of that median Euclidean distance.
To reduce variance when computing RBF CKA with small c , we use a minibatch size of 1000 for these experiments.
Figure J.1 shows the appearance of CKA heatmaps of a narrow, shallow network (ResNet-38 1 × , top), a wide network (ResNet-38 10 × , middle), and a deep network (ResNet-164 1 × , bottom).
Although heatmaps computed for a small network (ResNet-38 1 × ) look qualitatively similar regardless of kernels, both wide (ResNet-38 10 × ) and deep (ResNet-164 1 × ) networks exhibit signiﬁcant differences.
[Figure]
Because differences in representational similarity heatmaps ultimately reﬂect differences in the underlying kernel matrices in Figure J. we show kernel matrices of individual layers taken from inside the block structure of each network on random minibatches where the examples have been sorted in descending values of the ﬁrst principal component.
All kernels are sensitive to dominant datapoints but in different ways and to different degrees.
Linear kernel matrices are dominated by the similarity between dominant datapoints.
The cosine kernel ignores activation norms
[Figure]
and ﬁnds high similarity within groups of dominant and non-dominant datapoints but low similarity between groups.
The RBF kernel effectively considers all far away points to be equally dissimilar, and thus indicates that dominant datapoints are dissimilar to all other datapoints, including other dominant datapoints, which are typically far in Euclidean distance (because, while aligned in direction, they have different norms).
Note that the prevalence of dominant datapoints can differ across models and initializations, as previously demonstrated in Figure 4.
The dominant datapoints are clearly visible as a block in the top-left corner of the cosine kernel matrix.
For ResNet-38 10 × , there are 14 in the minibatch ofexamples that is shown
[Figure]
but for ResNet-164 ( 1 × ), there are only [Figure] Example 100 E x a m p l e Linear   1e
[Figure]
Example
Cosine  1e
[Figure]
What is the effect of removing dominant datapoints upon CKA similarity heatmaps computed with these other kernels?
In Figure J.3, we show that, once the dominant datapoints are removed from large networks, we again see only differences among CKA heatmaps computed with different kernels, in line with the results observed for shallow networks in Figure J.1.
There are no longer large blocks of many consecutive similar layers in any of the heatmaps.
Across all choices of kernel that we have investigated, when blocks appear in CKA heatmaps, they can be eliminated by eliminating the dominant datapoints.
[Figure]
ResNet-
×
[Figure]