Are Neurons Actually Collapsed? On the Fine-Grained Structure in Neural Representations
Anonymous Author(s
AffiliationAddress
Abstract
Recent work has observed an intriguing “Neural Collapse” phenomenon in welltrained neural networks, where the last-layer representations of training sampleswith the same label collapse into each other.
This suggests that the last-layerrepresentations are completely determined by the labels, and do not depend onthe intrinsic structure of input distribution.
We provide evidence that this is not acomplete description, and that the apparent collapse hides important fine-grainedstructure in the representations.
Specifically, even when representations apparentlycollapse, the small amount of remaining variation can still faithfully and accuratelycaptures the intrinsic structure of input distribution.
As an example, if we trainon CIFAR-10 using only 5 coarse-grained labels (by combining two classes intoone super-class) until convergence, we can reconstruct the original 10-class labelsfrom the learned representations via unsupervised clustering.
The reconstructedlabels achieve 93% accuracy on the CIFAR-10 test set, nearly matching the normal
CIFAR-10 accuracy for the same architecture.
Our findings show concretely howthe structure of input data can play a significant role in determining the fine-grainedstructure of neural representations, going beyond what Neural Collapse predicts.
Introduction
Much of the success of deep neural networks has, arguably, been attributed to their ability tolearn useful representations , or features , of the data (Rumelhart et al., 1985).
Although neuralnetworks are often trained to optimize a single objective function with no explicit requirementson the inner representations, there is ample evidence suggesting that these learned representationscontain rich information about the input data (Levy & Goldberg, 2014; Olah et al., 2017).
As a result,formally characterizing and understanding the structural properties of neural representations is ofgreat theoretical and practical interest and can provide insights on how deep learning works and howto make better use of these representations.
One intriguing phenomenon recently discovered by Papyan et al.
(2020) is Neural Collapse , whichidentifies structural properties of last-layer representations during the terminal phase of training(i e. after zero training error is reached).
The simplest of these properties is that the last-layerrepresentations for training samples with the same label collapse into a single point, which is referredto as “variability collapse (NC1).”
This is surprising, since the collapsed structure is not necessaryto achieve small training or test error, yet it arises consistently in standard architectures trained onstandard classification datasets.
A series of recent papers were able to theoretically explain Neural Collapse under a simplified modelcalled the unconstrained feature model or layer-peeled model (see Section 2 for a list of references).
In this model, the last-layer representation of each training sample is treated as a free optimizationvariable and therefore the training loss essentially has the form of a matrix factorization.
Under avariety of different setups, it was proved that the solution to this simplified problem should satisfy
Neural Collapse.
Although Neural Collapse is relatively well understood in this simplified model,this model completely ignores the role of the input data because the loss function is independent ofthe input data.
Conceptually, this suggests that Neural Collapse is only determined by the labelsand may happen regardless of the input data distribution.
Zhu et al.
(2021) provided further empiricalsupport of this claim via a random labeling experiment.
On the other hand, it is conceivable that the intrinsic structure of the input distribution shouldplay a role in determining the structure of neural net representations.
For example, if a class containsa heterogeneous set of input data (such as different subclasses), it is possible that their heterogeneityis also respected in their feature representations (Sohoni et al., 2020).
However, this appears tocontradict Neural Collapse, because Neural Collapse would predict that all the representationscollapse into the same point as long as they have the same class label.
This dilemma motivates us tostudy the following main question in this paper:
How can we reconcile the roles of the intrinsic structure of input distribution vs. the explicitstructure of the labels in determining the last-layer representations in neural networks?
Our methodology and findings.
Tostudy the above question, we designexperiments to manually create a mismatch between the intrinsic structure ofthe input distribution and the explicitlabels provided for training in standardclassification datasets and measure howthe last-layer representations behave inresponse to our interventions.
This allows us to isolate the effect of the input distribution from the effect of labels.
Our main findings are summarized below.
First, both the intrinsic structure of theinput distribution and the explicit labels provided in training clearly affectthe structure of the last-layer representations.
The effect of input distributionemerges earlier in training, while theeffect of labels appears at a later stage.
For example, for both Coarse CIFAR-10 and Fine CIFAR-10, at some point the representationsnaturally form 10 clusters according to the original CIFAR-10 labels (which comes from the intrinsicinput structure), even though 5 or 20 different labels are provided for training.
Later in training (after100% training accuracy is reached), the representations collapse into 5 or 20 clusters driven by theexplicit labels provided, as predicted by Neural Collapse.
Second, even after Neural Collapse has occurred according to the explicit label information, the seemingly collapsed representations corresponding to each label can still exhibit fine-grained structuresdetermined by the input distribution.
As an illustration, Figure 1 visualizes the representations fromthe last epoch of training a ResNet-18 on Coarse CIFAR-10.
While globally there are 5 separatedclusters as predicted by Neural Collapse, if we zoom in on each cluster, it clearly consists of twosubclusters which correspond to the original CIFAR-10 classes.
We also find that this phenomenonpersists even after a very long training period (e.g. 1,000 epochs), indicating that the effect of inputdistribution is not destroyed by that of the labels, at least not within a normal training budget.
To further validate our finding that significant input information is present in the last-layer representations despite Neural Collapse, we perform a simple Cluster-and-Linear-Probe (CLP) procedure onthe representations from ResNet-18 trained on Coarse CIFAR-10, in which we use an unsupervisedclustering method to reconstruct the original labels, and then train a linear classifier on top of theserepresentations using the reconstructed labels.
We find that CLP can achieve > 93% accuracy on theoriginal CIFAR-10 test set, matching the standard accuracy of ResNet-18, even though only 5 coarselabels are provided the entire time.
Takeaway.
While Neural Collapse is an intriguing phenomenon that consistently happens in welltrained neural networks, we provide concrete evidence showing that it is not the most comprehensive description of the behavior of last-layer representations in practice, as it fails to capture the possiblefine-grained properties determined by the intrinsic structure of the input distribution.
Related Work
The Neural Collapse phenomenon was originally discovered by Papyan et al.
and has led to aseries of further investigations.
A number of papers Fang et al. (2021); Lu & Steinerberger (2020); Wojtowytsch et al.
Mixon et al.
; Zhu et al.
; Ji et al.
; Han et al.
; Zhou et al.
; Tirer &
Bruna (2022); Yaras et al. (2022) studied a simplified “unconstrained feature model”, also known as“layer-peeled model”, and showed that Neural Collapse provably happens under a variety of settings.
This model treats the last-layer representations of all training samples as free optimization variables.
By doing this, the loss function no longer depends on the input data, and therefore this line of work isunable to capture any effect of the input distribution on the structure of the representations.
Ergen &
Pilanci (2021); Tirer & Bruna (2022); Weinan & Wojtowytsch (2022) considered more complicatedmodels but still did not incorporate the role of the input distribution.
Hui et al.
(2022) studied the connection of Neural Collapse to generalization and concluded that
Neural Collapse occurs only on the training set, not on the test set.
Galanti et al.
(2021) found that
Neural Collapse does generalize to test samples as well as new classes, and used this observation tostudy transfer learning and few-shot learning.
Sohoni et al.
(2020) observed that the last-layer representations of different subclasses within the sameclass are often separated into different clusters, and used this observation to design an algorithm forimproving group robustness.
However, it is unclear whether their observation happens in the Neural
Collapse regime, while we find that fine-grained structure in representations can co-exist with Neural
Collapse.
Furthermore, Sohoni et al.
(2020) looked at settings in which different subclasses havedifferent accuracies and attributed the representation separability phenomenon to this performancedifference.
On the other hand, we find that representation separability happens even when there is noperformance gap between different subclasses.
Preliminaries and Setup
This section introduces the notation used throughout this paper, as well as a more detailed explanationof Neural Collapse and a clarification of the experiment setup.
For a classification task dataset, we denote it by D = { ( x k , y k ) } nk =1 ,
where ( x k , y k ) ∈ R d ′ ×
[ C ] isa pair of input features and label, n is the number of samples, d ′ is the input dimension, and C is thenumber of classes.
For a given neural network, we denote the last-layer representation by H ∈ R n × d , i.e. the hiddenrepresentation before the final linear transformation, where d is the last-layer dimensionality.
Foran original class c ∈ [ C ] , we denote the number of samples in class c by n c , and the last-layerrepresentation of k -th sample in class c by h ( c ) k .
3.1 Preliminaries of Neural Collapse
Neural Collapse characterizes 4 phenomena, named NC1-NC4.
Here we introduce the first two whichconcern the structure of the last-layer representations.
NC1, or variability collapse, predicts that the variance of last-layer representations of samples withinthe same class vanishes as training proceeds.
Formally, it can be measured by NC 1 = 1 C Tr (cid:16) Σ W Σ † B (cid:17)(Papyan et al., 2020; Zhu et al., 2021), which should tend to 0 .
The Σ W and Σ B are defined asΣ W = 1 C (cid:88) c
∈ [ C ] n c n c (cid:88) i
= (cid:16) h
( c ) i − µ c (cid:17) (cid:16) h ( c )
i − µ c (cid:17) ⊤ and Σ B = 1 C (cid:88) c
∈ [ C ] ( µ
[Equation]
where µ c = 1 n c (cid:80) n c k =1 h ( c ) k are the class means and µ G = 1 n (cid:80) n k =1 h k is the global mean.
NC2 predicts that the class means converge to a special structure, i.e. their normalized covarianceconverges to the Simplex Equiangular Tight Frame (ETF).
This can be characterize by
[Equation]
during training, where M is the stack of centralized class-means, whose c -th row is µ c − µ G , and
C ∈ R C is an all-one vector in length C , I is the identity matrix.
3.2 Experiment Setup
In our experiment, we explore the role of input distribution and labels through assigning coarser orfiner labels to each sample, and then explore the structure of last-layer representation of a modeltrained on the dataset with coarse or fine labels and see to what extent the information of originallabels are preserved.
The coarse labels are created in the following way.
First choose a number ˜ C which is divisible by C ,and create coarse labels by ˜
y k
= y k mod ˜ C , which merges the classes whose index have the samemodulus w.r.t.
˜ C
and thus create ˜ C super-classes.
Since the original index of classes generally has nospecial meanings, this process should act similar to randomly merging classes 1 .
We say the sampleswith the same coarse label belongs to the same super-class, and call the dataset ˜ D
= { ( x k , ˜ y k ) } nk =
1the coarse dataset, which is the dataset we used to train the model.
To create fine labels, we randomly split each class into two sub-classes.
Specifically, the fine labelsare created by ˆ y k
= y k + βC , where ˆ y k is the fine label of sample k and C is the number of originalclasses and β is a Bernoulli Variable.
This process result in a dataset ˆ D = { ( x k , ˆ y k ) } n k =1 with 2 Cclasses.
Same as before, we call ˆ D the fine dataset.
Exploring the Fine-Grained Representation Structure with CoarseCIFAR-10
In this section, we experiment with coarsely labeled datasets, using Coarse CIFAR-10 as an illustrativeexample, and defer the experiments with fine labels to appendix.
Specifically, the model is trained onthe training set of Coarse CIFAR-10 for a certain number of steps that is sufficient for the model toconverge.
We then take the last-layer representations of the model during training and explore theirstructure.
In order to make an exhaustive observation, the experiment is repeated using different learning ratesand weight-decay rates.
Specifically, we choose the learning rate of training in { 10 − 1 , 10 − 2 , 10 − 3 }and weight-decay rate in { 5 × 10 − 3 , 5 × 10 − 4 , 5 × 10 − 5 } and permute all possible combinationsof them.
The experiment is also conducted on multiple datasets and network architectures.
Due tospace limit, in this section we focus on experiment of ResNet-18 on Coarse CIFAR-10, where theoriginal C = 10 and the number of coarse labels is chosen as ˜ C = 5 .
For training hyper-parameters,we only report results for two representative hyper-parameter combinations, referred to as config #1and config #2.
The learning rate of config #1 is set to 0 .
01 and weight-decay rate is 5 × 10 − 4 , and forconfig #2 the learning rate is 0 .
01 and weight decay rate is 5 × 10 − 5 .
Both settings reach 0 trainingerror quickly during training.
We defer complete experiment results to appendix.
As a preliminary result, we first show that under our experiment settings, Neural Collapse doeshappen, i.e. the representation does converge to 5 clusters and the class-means form a Simplex ETFstructure.
Specifically, we measure NC 1 and NC 2 defined in Section 3.1 for both config #1 and config#2, with C replaced by ˜ C since we are calculating it on coarse dataset.
The results are shown in
Figure 2, which matches previous results in Papyan et al.
Zhu et al.
, indicating that
Neural Collapse does happen under both settings.
4.1 Class Distance
We first look at the average square Euclidian distance of last-layer representations between eachtwo original classes.
Formally, we calculate a class distance matrix D ∈ R C × C , whose entries are
D i,j =
[Equation]
) u − h ( j ) v (cid:13)(cid:13)(cid:13) 2 2 , for all i, j ∈ [ C ] , where h ( c ) k represents the last-layerrepresentation of the k -th sample of super-class u .
Since the model is trained on coarse dataset, Neural Collapse asserts that for every original class pairi, j in the same super-class (including the case of i = j ), the class distance D i,j should be very small.
In Coarse CIFAR-10, this will result in three darks lines (let darker color represents lower value) inthe heatmap of D since each super class contains two original classes, as illustrated in Figure 3 (b).
For example, in Coarse CIFAR-10 the original class "airplane" and "dog" both belong to the superclass "airplane & dog", therefore per Neural Collapse’s prediction, their last-layer representationswould collapse to each other, making the average square distance extremely small compared to otherentries.
In contrast, if the last-layer representations perfectly reflects the distribution of input, i.e.original classes, the class distance matrix should be a diagonal matrix as shown in Figure 3 (a),because the last-layer representation of samples in each original class only collapse to the class-meanof this original class.
Figures 4 and 5 display the heatmap of class distance matrix D , arranged by number of trainingepochs.
From the results there are two surprising observations can be made: Firstly, there are indeedthree dark lines at the final stage, however, the three dark lines does not show up simultaneously, andthe central line – represents the samples in the same original classes – shows up earlier.
The secondobservation is, even in the final stage where the training error is zero, the three lines can be not of thesame degree of darkness, with the central line shallower, and this is especially apparent in Figure 5.
Those observations suggests that, the actual behaviour of the last-layer representations is betweenthe cases predicted in Figure 3 (a) and (b): the input distribution and training label both have animpact on the distribution of the last-layer representations, and although the time of them to show upis different, both of them can be present even after reaching zero training error for a long time.
Theseobservations suggest both the existence of Neural Collapse and the inadequacy of Neural Collapse tocompletely describe the behaviour of last-layer representations.
4.2 Visualization
In this subsection, we take a closer look at the last-layer representations of the model at the end oftraining by reducing the dimensionality of the last-layer representations to 2 through t-SNE (Van der
Maaten & Hinton, 2008) and visualize them.
Specifically, we visualize each super-class separately,but color the samples whose original labels are different with different colors.
Due to space limiation, we only show the visualization results of config #1 here in Figure 6 anddefer the complete result to appendix.
It can be observed that, there is distinguishable differencein the distribution of original labels in both config #1.
This suggests that, the input distributioninformation, i.e. the original label information, is well preserved in the last-layer representationsunder both settings, albeit for config #1 it seems it can not be distinguished through distance matrixin Section 4.1.
4.3 Learning CIFAR-10 from 5 Coarse Labels
As the results in Section 4.2 suggest, even after the training accuracy has reached 100% for a longtime, the samples within each super-class still exhibit a clear structure per their original class, andthose structures act as clusters after reducing dimensionality.
Inspired by this observation, we performa Cluster-and-Linear-Probe (CLP) test to evaluate to what extent the original class information ispreserved in the last-layer representations.
In CLP, we use the representations learned on Coarse
CIFAR-10 to reconstruct 10 labels and run a linear probe with the reconstructed labels.
Specifically,we first use t-SNE to reduce the dimensionality to 2 and then use KMeans to find 2 clusters of thedimensionality-reduced representations within each super-class.
We use the clusters as reconstructedlabels to do a linear probe.
In linear probe, we train a linear classifier on top of the previously learnedrepresentation H on the training set with reconstructed labels and evaluate the learned linear classifieron original test set.
Notice that because we don’t know the mapping of reconstructed classes to trueoriginal classes, we permute each possible mapping and report the highest performance.
We alsotrain a linear probe with original training labels as a comparison.
The results of config #1 and config#2 are shown in Figures 7 and 8 respectively.
In most cases, the performance of CLP on original test set is comparable with linear probe trained ontrue original labels or even with models originally trained on CIFAR-10.
Notice that the representation
H is obtained through the model trained with coarse labels, and the label reconstruction only usesinformation of H and the number of original classes.
This means we can achieve a very highperformance on original test set even if we only have access to almost only the information of coarselabels.
This result further confirms that the input distribution can play an important role in thelast-layer representations.
10 with the same training hyper-parameters of config #
Discussion
In this paper, we initiated the study of the role of the intrinsic structure of the input data distributionon the last-layer representations of neural networks, and in particular, how to reconcile it with the
Neural Collapse phenomenon, which is only driven by the explicit labels provided in the trainingprocedure.
Through a series of experiments, we provide concrete evidence that the representationscan exhibit clear fine-grained structure despite their apparent collapse.
While Neural Collapse is anintriguing phenomenon and deserves further studies to understand its cause and consequences, ourwork calls for more scientific investigations of the structure of neural representations that go beyond
Neural Collapse.
We note that the fine-grained representation structure we observed depends on the inductive biasesof the network architecture and the training algorithm, and may not appear universally.
In ourexperiments on Coarse CIFAR-10, we observe the fine-grained structure for ResNet and DenseNet,but not for VGG (see appendix for extended results).
We also note that certain choices of learningrate and weight-decay rate lead to stronger fine-grained structure than others.
We leave a thoroughinvestigation of such subtlety for future work.
A How Does Semantic Similarity Affect the Fine-Grained Structure?
In our experiments with Coarse CIFAR-10, each coarse label is obtained by combining two classesregardless of the semantics.
The fact that the neural network can separate the two classes in itsrepresentation space implies that the network recognizes these two classes as semantically different(even though they are given the same coarse label).
In this section, we explore the following question:
If the sub-classes in a super-class have semantic similarity, will the representations still exhibit afine-grained structure to distinguish them?
Intuitively, if the coarse label provided is “natural” andconsists of semantically similar sub-classes, it is possible that the neural network will not distinguishbetween them and just produce truly collapsed representations.
We take an initial step towards this question by looking at ResNet-18 trained on CIFAR-100 using theofficial 20 super-classes (each super-class contains 5 sub-classes) as labels.
Unlike randomly mergingclasses as we did in Section 4, the official super-classes of CIFAR-100 are natural, merging classeswith similar semantics (for example, “beaver” and “dolphin” both belong to “aquatic mammals”).
This offers a perfect testbed for our question.
We find that ResNet-18 indeed is not able to distinguish all sub-classes in its representation space, butcan still produce separable representations if some sub-classes within a super-class are sufficientlydifferent.
Interestingly, the notion of semantic similarity of ResNet-18 turns out to agree well withthat of humans.
Figures 9 and 10 show the t-SNE visualizations of representations from two superclasses.
From the visualizations, although there are not as clear clusters as for Coarse CIFAR-10,the representations do exhibit visible separations between certain sub-classes.
In Figure 9, “bicycles”and “motorcycles” are entangled together, while they are separated from “bus”, “pickup truck”, and“train”, which is human-interpretable.
In Figure 10, “crab” and “lobster” are mixed together, which areboth aquatic and belong to malacostraca, while the other three are not and have more differentiativerepresentations.
These results confirm the intuition that the fine-grained structure in last-layer representations isaffected by, or even based on, the semantic similarity between the inputs.
B Complete Coarse CIFAR-10 Experiment Results
From this section, we provide extended experiment results.
As mentioned in Section 3.2, we permutelearning rate in { 10 − 1 , 10 − 2 , 10 − 3 } and weight decay rate in { 5 × 10 − 3 , 5 × 10 − 4 , 5 × 10 − 5 } .
Generally, the results will be shown in a 3 × 3 table, of which each grid represents the result of onehyper-parameter combination, with each row has the same learning rate and each column has thesame weight decay rate.
In this section, we repeat the experiments in Section 4 with all learning rate and weight-decay ratecombinations.
Firstly, we present the training statistics (accuracy, loss) of all hyper-parameters in
Figures 11 and 12 as an reference.
It can be observed that all hyper-parameter groups achieved verylow training error except the first one (weight decay = 5 × 10 − 3 , learning rate = 10 − 1 ).
In fact, thelast two hyper-parameter combinations (learning rate = 10 − 3 , weight decay ∈ { 5 × 10 − 4 , 5 × 10 − 5 } )didn’t achieve exactly 0 training error (their training error are < 0 .
5% but not exactly 0 ), and theother 6 hyper-parameter combinations all achieved exactly 0 training error.
B.1 Class Distance
Here we present the visualization of the heatmap of class distance matrix D which is defined in
Section 4.1.
We choose 4 epochs to show the trend during training.
The results are presented in
Figures 13 to 16, whose epoch numbers are 20 , 120 , 240 and 349 respectively.
B.2 Visualization
In this section, we present the t-SNE visualization result of ResNet-18 on Coarse CIFAR-10 in
Figures 17 to 19.
The results are divided into three groups, each of which has the same learning rateand the format of each group is the same as Figure 6.
B.3 Cluster-and-Linear-Probe
The Cluster-and-Linear-Probe test results of ResNet-18 trained on Coarse CIFAR-10 with all hyperparameter combinations are presented in Figure 20.
C Class Distance Result of Fine CIFAR-10
In this section, we consider finely-labeled dataset.
We construct a fine version of CIFAR-10 withthe process described in Section 3.2, and call it Fine CIFAR-10.
We provide the visualization of theclass distance matrix of Fine CIFAR-10 with all hyper-parameter combinations.
As before, multipleepochs during training are selected to display a evolutionary trend of the class distance matrices.
Theresults are presented in Figures 21 to 23, whose epoch numbers are 20 , 200 and 350 respectively.
D Experiment of ResNet-18 on Coarse CIFAR-100
In this section, we report extended experiment results on Coarse CIFAR-100.
The same with the caseof Coarse CIFAR-10, we construct CIFAR-100 through the label coarsening process described in
Section 3.2 and choose ˜ C = 20 , so that every 5 original classes are merged into one super-class.
Werepeat most of experiments in Section 4.
D.1 Class Distance
The heatmaps of distance matrices are presented in Figures 24 to 26, whose epoch number are 20 , 200and 350 respectively.
D.2 Cluster-and-Linear-Probe
Notice that we omit the visualization result of Coarse CIFAR-100 since there are too many figures.
We present the Cluster-and-Linear-Probe results to reflect the clustering property of last-layerrepresentations learned on Coarse CIFAR-100.
The CLP results are presented in Figure 27.
E Random Coarse CIFAR-10
In this section, as mentioned in Section 3.2, we make our experiment more complete by performing arandom combination of labels on CIFAR-10 rather than using a determined coarsening process as inthe main paper.
The dataset construction is almost the same as the process of assigning coarse labelsdescribed in Section 3.2, except here we randomly shuffle the class indices before coarsening them.
The class distance matrices of three difference epochs are shown in Figures 28 to 30.
From the resultswe can see, although there are no longer three dark lines, for each row there are generally two darkblocks, represents the original classes belongs to the same super-class, and the same observations in
Section 4 can still be made here.
F Experiment with DenseNet
We also perform our experiments with different neural network structures for completeness.
Inthis section, we show the result with DenseNet-121 on Coarse CIFAR-10.
The experiments with
DenseNet is supportive to our observations in the main paper.
F.1 Class Distance
The class distance matrices of three epochs during training are presented in Figures 31 to 33.
F.2 Cluster-and-Linear-Probe
The Cluster-and-Linear-Probe test results are presented in Figure 34.
G Experiment with VGG
We also extend our experiments to VGG-18.
Interestingly, VGG to some extent is a counter exampleof the observations made in the main paper: it only displays Neural Collapse, and can not distinguishdifferent original classes within one super-class, even in an early stage of training.
The reason why
VGG is abnormal requires further exploration.
The class distance matrices of three epochs with VGG during training are shown in Figures 35 to 37.
It can be observed that the three dark lines appears almost at the same time and always be of nearlythe same darkness.
This represents the trend predicted by Neural Collapse (Figure 3 (a)), but rejectsthe prediction made by (Figure 3 (b)).