N EURAL C OLLAPSE I NSPIRED F EATURE -C LASSIFIER A LIGNMENT FOR F EW - SHOT C LASS I NCREMENTAL
L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
Few-shot class-incremental learning (FSCIL) has been a challenging problem as only a few training samples are accessible for each novel class in the new sessions.
Finetuning the backbone or adjusting the classiﬁer prototypes trained in the prior sessions would inevitably cause a misalignment between the feature and classiﬁer of old classes, which explains the well-known catastrophic forgetting problem.
In this paper, we deal with this misalignment dilemma in FSCIL inspired by the recently discovered phenomenon named neural collapse , which reveals that the last-layer features of the same class will collapse into a vertex, and the vertices of all classes are aligned with the classiﬁer prototypes, which are formed as a simplex equiangular tight frame (ETF).
It corresponds to an optimal geometric structure for classiﬁcation due to the maximized Fisher Discriminant Ratio.
We propose a neural collapse inspired framework for FSCIL.
A group of classiﬁer prototypes are pre-assigned as a simplex ETF for the whole label space, including the base session and all the incremental sessions.
During training, the classiﬁer prototypes are not learnable, and we adopt a novel loss function that drives the features into their corresponding prototypes.
Theoretical analysis shows that our method holds the neural collapse optimality and does not break the feature-classiﬁer alignment in an incremental fashion.
Experiments on the miniImageNet, CUB-200, and
CIFAR-100 datasets demonstrate that our proposed framework outperforms the state-of-the-art performances.
Our code will be publicly available.
I NTRODUCTION
Human perception is easy to learn new classes continually with only limited samples, while for machine learning models, it is still a challenging task despite the great success in classiﬁcation of a closed label space (LeCun et al., 2015).
However, learning incrementally and learning with few-shot labeled samples are common in the real-world implementations, and in many applications, such as robotics, the two demands emerge simultaneously.
To this end, few-shot class-incremental learning (FSCIL) was proposed to tackle this problem (Tao et al., 2020b).
Compared with few-shot learning (Ravi & Larochelle, 2017; Vinyals et al., 2016), FSCIL transfers a trained model into new label spaces incrementally.
It also differs from incremental learning (Cauwenberghs & Poggio, 2000; Li & Hoiem, 2017; Rebufﬁ et al., 2017) in that there are only a few (usually 5) samples accessible for each new class in the incremental sessions.
For each session’s evaluation, the model is required to infer test images coming from all the classes that have been encountered.
The base session of FSCIL contains a large label space and sufﬁcient training samples, while each incremental session only has a few novel classes and labeled images.
It poses the notorious catastrophic forgetting problem (Goodfellow et al., 2013) because the novel sessions have no access to the data of the previous sessions.
Due to the importance and difﬁculty, FSCIL has attracted much research attention.
The initial solutions to FSCIL ﬁnetune the network on new session data with distillation schemes to reduce the forgetting of old classes (Tao et al., 2020b; Dong et al., 2021).
However, the few-shot data in novel sessions can easily induce over-ﬁtting.
Following studies favor training a backbone network on the base session as a feature extractor (Zhang et al., 2021; Hersche et al., 2022; Aky¨urek et al., 2022). [Figure]
As a comparison, we pre-assign and ﬁx an optimal feature-classiﬁer alignment, and then train a model towards the same neural collapse optimality in each session to avoid target conﬂict.
For novel sessions, the backbone network is ﬁxed and a group of novel-class prototypes (classiﬁer vectors) are learned incrementally.
But as shown in Figure 1 (a), the newly added prototypes may lie close to the old-class prototypes, which impedes the ability to discriminate between the old-class and the novel-class samples in evaluation.
As a result, adjusting the classiﬁer prototypes is always necessary for two goals: ( i ) keep a sufﬁcient distance between the old-class and the novel-class prototypes; ( ii ) prevent the adjusted old-class prototypes from shifting far away from their original positions.
However, the two goals rely on sophisticated loss functions or regularizers (Chen & Lee, 2021; Hersche et al., 2022; Aky¨urek et al., 2022), and are hard to attain simultaneously without qualiﬁcation.
Besides, as shown in Figure 1 (a), there will be a misalignment between the adjusted classiﬁer and the ﬁxed features of old classes.
A recent study proposes to reserve feature space for novel classes to circumvent their conﬂict with old classes (Zhou et al., 2022a), but an optimal feature-classiﬁer alignment is hard to be guaranteed with learnable classiﬁer (Pernici et al., 2021).
We point out that it is the misalignment dilemma between feature and classiﬁer that causes the catastrophic forgetting problem of old classes.
If a backbone network is ﬁnetuned in novel sessions, the features of old classes will be easily deviated from their classiﬁer prototypes.
Alternatively, when a backbone network is ﬁxed and a group of new prototypes for novel classes are learned incrementally, the adjustment of old-class prototypes will also induce misalignments with their ﬁxed features.
In this paper, we pose and study the following question, “Can we look for and pre-assign an optimal feature-classiﬁer alignment such that the model is optimized towards the ﬁxed optimality, so avoids conﬂict among sessions?”
1. M OTIVATIONS AND C ONTRIBUTIONS
Neural collapse is a recently discovered phenomenon that at the terminal phase of training (aftertraining error rate), the last-layer features of the same class will collapse into a single vertex, and the vertices of all classes will be aligned with their classiﬁer prototypes and be formed as a simplex
equiangular tight frame (ETF) (Papyan et al., 2020).
A simplex ETF is a geometric structure of K vectors in R d , d ≥ K − 1 .
All vectors have the same (cid:96) 2 norm of 1 and any pair of two different vectors has an inner product of − 1 K − 1 , which corresponds to the largest possible angle of K equiangular vectors.
Particularly when d = K − 1 , a simplex ETF reduces to a regular simplex such as triangle and tetrahedron.
It describes an optimal geometric structure for classiﬁcation due to the minimized within-class variance and the maximized between-class variance (Martinez & Kak, 2001), which indicates that the Fisher Discriminant Ratio (Fisher, 1936; Rao, 1948) is maximized.
Following studies aim to theoretically explain this phenomenon (Fang et al., 2021; Han et al., 2022).
It is expected that imperfect training condition, such as imbalance, cannot induce neural collapse and will cause deteriorated performance (Fang et al., 2021; Yang et al., 2022b).
Training in an incremental fashion will also break the neural collapse optimality.
Since neural collapse offers us an optimal structure where features and their classiﬁer prototypes are aligned, we can pre-assign such a structure and learn the model towards the optimality.
Inspired by this insight, in this paper, we initialize a group of classiﬁer prototypes ˆ
W ETF ∈ R d × ( K 0 + K (cid:48) ) as a simplex ETF for the whole label space, where K 0 is the number of classes in the base session and K (cid:48) is the number of classes in all the incremental sessions.
As shown in Figure 1 (b), it serves as the optimization target and keeps ﬁxed throughout all sessions training.
We append a projection layer after the backbone network and store the mean latent feature of each class output by the backbone in a memory.
In the training of incremental sessions, we only ﬁnetune the projection layer using a novel loss function that drives the ﬁnal features towards their corresponding target prototypes.
Without bells and whistles, our method achieves superior performances and relieves the catastrophic forgetting problem.
The contributions of this paper can be summarized as follows:
• To relieve the misalignment dilemma in FSCIL, we propose to pre-assign an optimal alignment inspired by neural collapse as a ﬁxed target throughout the incremental learning. Our model is trained towards the same optimality to avoid optimization conﬂict among sessions.
• We ﬁx the prototypes and apply a novel loss function that only ﬁnetunes a projection layer to drive the output features into their corresponding prototypes.
Theoretical and empirical analyses show that our method better holds the neural collapse optimality.
• Experiments on miniImageNet, CIFAR-100, and CUB-200 demonstrate that our method is able to surpass the state-of-the-art performances.
In particular, our method achieves an average accuracy improvement of more than 3.5% over a recent strong baseline on both miniImageNet and CIFAR-100.
R ELATED W ORK
Few-shot class-incremental learning (FSCIL).
As a variant of class-incremental learning (CIL) (Cauwenberghs & Poggio, 2000; Li & Hoiem, 2017; Rebufﬁ et al., 2017), FSCIL only has a few novel-classes and training data in each incremental session (Tao et al., 2020b;
Dong et al., 2021), which increases the tendency of overﬁtting on novel classes (Snell et al., 2017; Sung et al., 2018).
Both CIL and FSCIL require a delicate balance between well adapting a model to novel classes and less forgetting of old classes (Zhao et al., 2021).
A popular choice is to use meta learning (Yoon et al., 2020; Chi et al., 2022;
Zhou et al., 2022b).
Some studies try to make base and incremental sessions compatible via pseudo-feature (Cheraghian et al., 2021b;
Zhou et al., 2022a), augmentation (Peng et al., 2022), or looking for a ﬂat minima (Shi et al., 2021).
For training in incremental sessions, the new prototypes for novel classes should be separable from the old-class prototypes.
Meanwhile, the adjustment of old-class prototypes should not induce large shifts.
Current studies widely rely on evolving the prototypes (Zhang et al., 2021; Zhu et al., 2021a) or sophisticated designs of loss and regularizer (Ren et al., 2019;
Hou et al., 2019; Tao et al., 2020a; Joseph et al., 2022; Lu et al., 2022;
Hersche et al., 2022; Chen & Lee, 2021;
Aky¨urek et al., 2022; Yang et al., 2022a).
However, the two goals have inherent conﬂict, and a tough effort to balance the loss terms is necessary.
In contrast, our method pre-assigns and ﬁxes a feature-classiﬁer alignment as an optimality.
A model is trained towards the same target in all sessions.
We only use a single loss without any regularizer .
Neural collapse.
First discovered in Papyan et al.
(2020), neural collapse describes an elegant geometric structure of the last-layer feature and classiﬁer in a trained model.
It inspires later studies to theoretically explain this phenomenon.
Based on a simpliﬁed model that only considers the lastlayer optimization, neural collapse is proved to be the global optimality of training on balanced data with the CE (Weinan & Wojtowytsch, 2020; Graf et al., 2021; Lu & Steinerberger, 2020; Fang et al., 2021; Zhu et al., 2021b;
Ji et al., 2022) and the MSE (Mixon et al., 2020; Poggio & Liao, 2020;
Zhou et al., 2022c; Han et al., 2022; Tirer & Bruna, 2022) loss functions.
Recent studies try to induce neural collapse in imbalanced training by ﬁxing a classiﬁer or novel loss (Yang et al., 2022b;
Xie et al., 2022).
Our method is inspired by Yang et al.
(2022b),
but we apply the classiﬁer in an incremental fashion.
Galanti et al.
(2022) show that neural collapse is still valid when transferring a model into new samples or classes.
To the best of our knowledge, we are the ﬁrst to study FSCIL from the neural collapse perspective, which offers our method sound interpretability .
B ACKGROUND 3. F EW -S HOT C LASS -I NCREMENTAL L EARNING (FSCIL)
In real-world applications, one often needs to adapt a model to data coming from a new label space with only a few labeled samples.
FSCIL trains a model incrementally on a sequence of training datasets {D (0) , D (1) , . . .
, D ( T ) } , where D ( t ) = { ( x i , y i ) } |D ( t )
| i
=1 , D (0) is the base session, and T the number of incremental sessions.
The base session D (0) usually contains a large label space C (0) and sufﬁcient training images for each class c ∈ C (0) .
In each incremental session D ( t ) , t > 0 , there are only a few labeled images and we have |D ( t ) |
= pq , where p is the number of classes and q is the number samples per novel class, known as p -way q -shot.
The label space C ( t ) has no overlap with any other session, i.e., C ( t ) ∩ C ( t (cid:48) )
= ∅ , ∀ t (cid:48)
(cid:54) =
t .
For any incremental session t > 0 , we only have access to the data in D ( t ) , and the training sets of the previous sessions are not available.
For evaluation in session t , the test dataset comes from all the encountered classes in the previous and current sessions 1 , i.e. the label space of ∪ ti =
C
( i ) .
Therefore, FSCIL suffers from severe data scarcity and imbalance.
It requires a model to be adaptable to novel classes, and meanwhile keep the ability on old classes.
3. N EURAL C OLLAPSE
Neural collapse refers to a phenomenon at the terminal phase of training (after 0 training error rate) on balanced data (Papyan et al., 2020).
It reveals a geometric structure formed by the last-layer feature and classiﬁer that can be deﬁned as:
Deﬁnition 1 (Simplex Equiangular Tight Frame)
A simplex Equiangular Tight Frame (ETF) refers to a matrix that is composed of K vectors in R d and satisﬁes:
[Equation]
where E = [ e 1 , · · · , e K ] ∈ R d × K , U ∈ R
d
×
K allows a rotation and satisﬁes U T U
=
I
K , I K is the identity matrix, and 1 K is an all-ones vector.
All column vectors in E have the same (cid:96) 2 norm and any pair has an inner produce of − 1 K − 1 , i.e.,
[Equation]
where δ k 1 ,k 2 = 1 when k 1 = k 2 , and 0 otherwise.
The neural collapse phenomenon includes the following four properties: (NC1) :
The last-layer features of the same class will collapse into their within-class mean, i.e., the covariance Σ ( k ) W → 0 , where Σ ( k ) W =
Avg i { ( µ k,i − µ k )( µ k,i − µ k ) T } , µ k,i is the feature of sample i in class k , and µ k is the within-class mean of class k features; (NC2) :
The within-class means of all classes centered by the global mean will converge to the vertices of a simplex ETF deﬁned in Deﬁnition 1, i.e., ˆ µ k , 1 ≤ k ≤ K satisfy Eq.
(2), where ˆ µ k = ( µ k − µ G ) / (cid:107) µ k − µ G (cid:107) and µ G is the global mean; (
NC
The within-class means centered by the global mean will be aligned with (parallel to) their corresponding classiﬁer weights, which means the classiﬁer weights will converge to the same simplex ETF, i.e., ˆ µ k = w k / (cid:107) w k (cid:107) , 1 ≤ k ≤ K , where w k is the classiﬁer weight of class k ; (NC4) :
When (NC1) - (NC3) hold, the model prediction using logits can be simpliﬁed to the nearest class centers 2 , i.e., arg max k (cid:104) µ , w k (cid:105) =
arg min k || µ − µ k
|| , where (cid:104)·(cid:105) is the inner product operator, µ is the last-layer feature of a sample for prediction.
Neural collapse corresponds to an optimal feature-classiﬁer alignment for classiﬁcation due to the maximized Fisher Discriminant Ratio (between-class variance to within-class variance).
M ETHOD
Neural collapse tells us an optimal geometric structure for classiﬁcation problems where the lastlayer features and classiﬁer prototype of the same class are aligned, and those of different classes are maximally separated.
However, this structure will be broken in imperfect training conditions, such as imbalanced training data (Fang et al., 2021; Yang et al., 2022b).
As illustrated in Figure(a), training in an incremental fashion will also break the neural collapse optimality.
Inspired by this perspective, what we should do for FSCIL is to keep the neural collapse inspired feature-classiﬁer alignment as sound as possible.
Concretely, we adopt a ﬁxed classiﬁer and a novel loss function as described in Section 4.1 and Section 4.2, respectively.
We introduce our framework for FSCIL in
Section 4.3.
Finally, in Section 4.4, we conduct theoretical analysis to show how our method better holds the neural collapse optimality in an incremental fashion.
4. ETF C LASSIFIER
Assume that the base session contains a label space of K 0 classes, each incremental session has p classes, and we have T incremental sessions in total.
The whole label space of this FSCIL problem has K 0 + K (cid:48) classes, where K (cid:48) =
Tp , i.e., we need to learn a model that can recognize samples from K
+ K (cid:48) classes.
We denote a backbone network as f , and then we have µ = f ( x , θ f ) , where µ ∈ R d is the output feature of input x , and θ f is the backbone network parameters.
A popular choice in current studies learns f and W (0) using the base session data, where W (0) ∈
R d × Kis the classiﬁer prototypes for base classes.
In incremental sessions t > 0 , f is ﬁxed as a feature extractor and only W ( t ) ∈ R
d × p for novel classes is learnable.
As shown in Figure 1 (a), one need to adjust { W (0) , · · · , W ( t ) } via sophisticated loss or regularizer to ensure separation among these prototypes (Aky ¨ urek et al., 2022; Hersche et al., 2022).
But it will inevitably introduce misalignment between the adjusted prototypes and the ﬁxed features of old classes.
It is an underlying reason for the catastrophic forgetting problem (Joseph et al., 2022).
Since neural collapse describes an optimal geometric structure of the last-layer feature and classiﬁer, we pre-assign such an optimality by ﬁxing a learnable classiﬁer as the structure instructed by neural collapse.
Following Yang et al.
(2022b)
, we adopt an ETF classiﬁer that initializes a classiﬁer as a simplex ETF and ﬁxes it during training.
The difference lies in that we use it in an incremental fashion.
Concretely, we randomly initialize classiﬁer prototypes
ˆ W ETF ∈ R d × ( K 0 + K (cid:48) ) by Eq.
(1) for the whole label space, i.e., the union of classes in all session, ∪ Ti =
C ( i ) .
We have K 0 =
|C (0) | and K (cid:48)
= (cid:80)
Ti =1 |C
( i ) | =
Tp .
Then any pair ( k 1 , k 2 ) of classiﬁer prototypes in ˆ W ETF satisﬁes:
[Equation]
[Equation]
where ˆ w k 1 and ˆ w k 2 are two column vectors in ˆ W ETF .
Our ETF classiﬁer ensures that the prototypes of the whole label space have the maximal pair-wise separation.
It serves as a ﬁxed target along the incremental training to avoid conﬂict among sessions.
We only need to learn a model whose output features are aligned with this pre-assigned structure.
4. D OT -R EGRESSION L OSS
The gradient of cross entropy (CE) loss with respect to the last-layer feature is composed of a pull term that drives the feature into its classiﬁer prototype of the same class, and a push term that pushes it away from the prototypes of different classes.
As pointed out by Yang et al. (2022b), when the classiﬁer prototypes are ﬁxed as an optimality, the pull term is always accurate towards the solution, and we can drop the push gradient that may be inaccurate.
Accordingly, we adopt a novel loss named dot-regression (DR) loss that can be formulated as (Yang et al., 2022b):
[Equation]
(4) where ˆ µ i is the normalized feature, i.e., ˆ µ i = µ i / (cid:107) µ i (cid:107) , µ i =
f
( x i , θ f ) , y i is the label of input x
i , ˆ w y i is the ﬁxed prototype in ˆ W ETF for class
y i , and we have (cid:107) ˆ
w y
i (cid:107) = 1 by Eq.
The total loss is an average over a batch of input x i .
The gradient of Eq.
(4) with respect to ˆ µ i takes the form of:
[Figure]
∂ L /∂ ˆ µ i = − (1 − cos ∠ (ˆ µ i , ˆ w y i ))
ˆ w y i .
It is shown that the gradient pulls feature ˆ µ i towards the direction of ˆ w y i , which is a pre-assigned target prototype.
Finally, the converged features will be aligned with ˆ W ETF , and thus the geometric structure instructed by neural collapse is attained.
The theoretical advantage of the DR loss has been proved in Yang et al.
(2022b).
In experiments, we will compare the DR loss with the CE loss to show its effectiveness in FSCIL.
4. NC-FSCIL
Based on the ETF classiﬁer and the DR loss, we now introduce our neural collapse inspired framework for few-shot class-incremental learning (NC-FSCIL).
As shown in Figure 2, our model is composed of two components, a backbone network f and a projection layer g .
The backbone network f takes the training data
x i as input, and outputs an intermediate feature
h
i .
The projection layer g can be a linear transformation or an MLP block following Hersche et al.
; Peng et al.
It projects the intermediate feature h i into µ i .
Finally, we perform an (cid:96) 2 normalization on µ i to get the output feature ˆ µ i , i.e.,
[Equation] [Equation]
where θ f and θ g denote the parameters of the backbone network and the projection layer, respectively.
We use the normalized output feature ˆ µ i to compute error signal by Eq.
In the base session t = 0 , we jointly train both f and g using the base session data.
The empirical risk to minimize in the base session can be formulated as:
[Equation]
where ˆ W ETF is the pre-assigned ETF classiﬁer as introduced in Section 4.1, L is the DR loss as introduced in Section 4.2, and ˆ µ i is a function of f and g as shown in Eq.
In each incremental session 1 ≤ t ≤ T , we ﬁx the backbone network f as a feature extractor, and only ﬁnetune the projection layer g .
As a widely adopted practice in FSCIL studies, a small memory of samples or features of old classes can be retained to relieve the overﬁtting on novel classes (Cheraghian et al., 2021a; Chen & Lee, 2021; Aky¨urek et al., 2022; Hersche et al., 2022).
Following Hersche et al.
, we only keep a memory M ( t ) of the mean intermediate feature h c for each old class c .
Concretely, we have,
[Equation]
where f has been ﬁxed after the base session.
Then we use D ( t ) as the input of f , and M ( t ) as the input of g to ﬁnetune the projection layer g .
The empirical risk to minimize in incremental sessions can be formulated as:
[Equation]
where ˆ µ i , and ˆ µ c are the output features of x i and h c , respectively, |D ( t ) | is the number of training samples in session t , and we have |M ( t ) | = (cid:80) t − 1 j =
|C ( j ) | .
Thanks to our pre-assigned alignment, we do not rely on any regularizer in our training .
In the evaluation of session t , we predict an input x based on the inner product between its output feature ˆ µ and the ETF classiﬁer prototypes:
[Equation]
4. T HEORETICAL S UPPORTS
We perform our theoretical work based on a simpliﬁed model that drops the backbone network and only keeps the last-layer features and classiﬁer prototypes as independent optimization variables.
This simpliﬁcation has been widely adopted in prior studies to facilitate analysis (Graf et al., 2021;
Fang et al., 2021; Zhu et al., 2021b).
We investigate the neural collapse optimality of an incremental problem of T sessions with our ETF classiﬁer.
Concretely, we consider the following problem,
[Equation]
where m ( t ) k,i ∈
R
d denotes a feature variable that belongs to the i -th sample of class k in session t , n k is number of samples in class k , K ( t ) is number of classes in session t , N ( t ) is the number of samples in session t , i.e., N ( t )
=
(cid:80) K ( t ) k
=1 n k , and M ( t ) ∈ R d × N ( t ) denotes a collection of m ( t ) k,i .
ˆ W ETF ∈ R d × K refers to the ETF classiﬁer for the whole label space as introduced in Section 4.1, and we have K =
(cid:80) Tt =
0 K ( t ) .
L can be both the cross entropy and the dot regression loss functions.
Theorem 1 Let ˆ M ( t ) denote the global minimizer of Eq.
(9) by optimizing the model incrementally from t = 0 , and we have ˆ M =
[ ˆ M (0) , · · · , ˆ M ( T ) ] ∈ R d × (cid:80) Tt =
N ( t ) .
When L in Eq.
(9) is CE or DR loss, for any column vector
ˆ m k
i in ˆ M whose class label is k , we have:
[Equation] [Equation]
,i
[Equation]
where K = (cid:80) Tt =
0 K ( t ) denotes the total number of classes of the whole label space, δ k,k (cid:48)
=
when k = k (cid:48) and 0 otherwise, and ˆ w k (cid:48) is the prototype of class k (cid:48) in ˆ W ETF .
The proof of Theorem 1 can be found in Appendix A. Eq.
(10) indicates that the global minimizer ˆ
M of Eq.
(9) satisﬁes the neural collapse condition, i.e., features of the same class collapse into a single vertex, and the vertices of all classes are aligned with ˆ W ETF as a simplex ETF.
It is shown that the feature space is equally separated by prototypes of all classes.
More importantly, in problem
Eq.
(9), the number of classes K ( t ) among T + 1 sessions and the number of samples n k among K classes can be imbalanced, which corresponds to the challenging demand of FSCIL.
E XPERIMENTS
In this section, we test our method on FSCIL benchmark datasets including miniImageNet (Russakovsky et al., 2015), CIFAR-100 (Krizhevsky et al., 2009), and CUB-200 (Wah et al., 2011), and compare it with state-of-the-art methods.
We also perform ablation studies to validate the effects of
ETF classiﬁer and DR loss.
Finally, we show the feature-classiﬁer structure achieved by our method.
5. I MPLEMENTATION D ETAILS
Please refer to Appendix B for our implementation details.
[Table] [Table]
5. P ERFORMANCE ON B ENCHMARKS
Our experiment results on minImageNet, CIFAR-100, and CUB-200 are shown in Table 1, Table 2, and Table 4 (Appendix C), respectively.
We see that our method achieves the best performance in all sessions on both miniImageNet and CIFAR-100 compared with previous studies.
ALICE (Peng et al., 2022) is a recent study that achieves strong performances on FSCIL.
Compared with this challenging baseline, we have an improvement of 2.61% in the last session on miniImageNet, and 2.01% on CIFAR-100.
We achieve an averaged accuracy improvement of more than 3.5% on both miniImageNet and CIFAR-100.
Although we do not surpass ALICE in the last session on CUB-200, we still have the best average accuracy among all methods.
As shown in the last rows of Table 1 and
Table 2, the improvement of our method lasts and even becomes larger in the ﬁrst several sessions.
It indicates that our method is able to hold the superiority and relieve the forgetting of old sessions.
5. A BLATION S TUDIES
We consider three models to validate the effects of ETF classiﬁer and DR loss.
All three models are based on the same framework introduced in Section 4.3 including the backbone network, the projection layer, and the memory module.
The ﬁrst model uses a learnable classiﬁer and the CE [Table] [Table] [Figure]
[Table] [Figure]
[Table]
[Figure]
[Table]
[Figure]
loss, which is the most adopted practice.
The second model only replaces the classiﬁer with our
ETF classiﬁer and also uses the CE loss.
The third model corresponds to our method using both
ETF classiﬁer and DR loss.
As shown in Table when a ﬁxed ETF classiﬁer is used, the ﬁnal session accuracies are signiﬁcantly better and the performance drops get much mitigated.
Adopting the DR loss is able to further moderately improve the performances.
It indicates that the success of our method is largely attributed to ETF classiﬁer and DR loss, as they pre-assign a neural collapse inspired alignment and drive a model towards the ﬁxed optimality, respectively.
5. F EATURE -C LASSIFIER S TRUCTURE
We check the feature-classiﬁer alignment instructed by neural collapse using our method and “Learnable+CE” as a comparison.
As shown in Figure 3, the average cosine similarities between features and classiﬁer prototypes of different classes, i.e., Avg k (cid:54) =
k (cid:48) { cos ∠ ( m k − m G , w k (cid:48) ) } , of our method are consistently lower than those of the baseline.
Most values of our method are negative and close to 0, which is in line with the guidance from neural collapse as derived in Eq.
Particularly in Figure 3b and Figure 3d, the average cosine similarities between m k − m G and w k (cid:48)
( k (cid:54) =
k (cid:48) )
among all encountered classes increase fast with session for the baseline method, while ours keep relatively ﬂat.
It indicates that the baseline method reduces the feature-classiﬁer margin of different classes as training incrementally, and our method enjoys a stable alignment.
As shown in Figure 4 and Figure 5, we also calculate the average cosine similarities between feature and classiﬁer of the same class, i.e., Avg k { cos ∠ ( m k − m G , w k ) } , and the trace ratio of within-class covariance to between-class covariance, tr(Σ W ) / tr(Σ B ) .
These results together support that our method better holds the feature-classiﬁer alignment and relieves the forgetting problem.
C ONCLUSION
In this paper, we propose to ﬁx a learnable classiﬁer as a geometric structure instructed by neural collapse for FSCIL.
It pre-assigns an optimal feature-classiﬁer alignment as a ﬁxed target throughout incremental training, which avoids optimization conﬂict among sessions.
Accordingly, a novel loss function that drives features towards this pre-assigned optimality is adopted without any regularizer.
Both theoretical and empirical results support that our method is able to hold the alignment in an incremental fashion, and thus relieve the forgetting problem.
In experiments of FSCIL, we achieve and even surpass the state-of-the-art performances on three datasets.
S TATEMENTS
Ethics Statement.
Our study does NOT involve any of the potential issues such as human subject, public health, privacy, fairness, security, etc .
All authors of this paper conﬁrm that they adhere to the ICLR Code of Ethics.
Reproducibility Statement.
For our theoretical result Theorem 1, we offer the proof in Appendix
A. All datasets used in this paper are public and have been cited.
Please refer to Appendix B for the dataset descriptions and the implementation details of our experiments.
Our source code will be publicly available in the ﬁnal version.
A A PPENDIX : P ROOF OF T HEOREM
Our proof is following Yang et al.
(2022b).
We consider the problem in Eq.
[Equation]
where m ( t ) k,i ∈
R
d denotes a feature variable that belongs to the i -th sample of class k in session t , n k is number of samples in class k , K ( t ) is number of classes in session t , N ( t ) is the number of samples in session t , i.e., N ( t )
=
(cid:80) K ( t ) k
=1 n k , and M ( t ) ∈ R d × N ( t ) denotes a collection of m ( t ) k,i .
ˆ W ETF ∈ R d × K refers to the ETF classiﬁer for the whole label space as introduced in Section 4.1.
We have K =
(cid:80) Tt =
0 K ( t ) and,
[Equation]
where ˆ w k and ˆ w k (cid:48) are two column vectors in ˆ W ETF .
From the deﬁnition of a simplex ETF in Eq.
(1), we have ˆ W ETF · 1 K = 0 d , where 1 K is an all-ones vector in R K , and
0 d is an all-zeros vector in R d .
Then we have,
[Equation] [Equation]
When L is the dot-regression (DR) loss in Eq.
(4), it is easy to identify that L ≥ 0 and the equality holds if and only if ˆ w Tk m ( t ) k,i = 1 , ∀ 0 ≤ t ≤ T, 1 ≤ k ≤ K, 1 ≤ i ≤ n k .
Since (cid:107) ˆ w k (cid:107) =
and (cid:107) m ( t ) k,i (cid:107) 2 ≤ 1 , we have ˆ w Tk m ( t ) k,i ≤ 1 .
The equality holds if and only if (cid:107) m ( t ) k,i (cid:107) 2 = 1 and cos ∠ ( ˆ w k , m ( t ) k,i ) = 1 .
Denote ˆ M =
[ ˆ M (0) , · · · , ˆ M ( T ) ] ∈ R d × (cid:80) Tt =
N ( t ) as the global optimality of
Eq.
(9) for all sessions 0 ≤ t ≤ T .
For any column vector ˆ m k,i in ˆ M , we have,
[Equation]
which concludes the proof for DR loss.
When L is the cross-entropy (CE) loss, i.e.,
[Equation]
where 0 ≤ t ≤ T, 1 ≤ k ≤ K ( t ) , and 1 ≤ i ≤ n k .
Since the problem is separable among T +sessions, we only analyze the t -th session and omit the superscript ( t ) for simplicity.
The objective in Eq.
(13) is the sum of an afﬁne function and log-sum-exp functions.
When ˆ W ETF is ﬁxed, the loss is convex w.r.t m k,i with convex constraints.
So, we can use the KKT condition for its global optimality.
Based on Eq.
(9) and Eq.
(13), we have the Lagrange function,
[Equation]
where λ k,i is the Lagrange multiplier.
The gradient with respect to m k,i takes the form of:
[Equation]
where 1 ≤ i ≤ n k , 1 ≤ k ≤ K ( t ) , and p j is the softmax probability of m k,i for the j -th class, i.e.,
[Equation]
[Equation]
Since (cid:107) ˆ w k (cid:107) = 1 and (cid:107) m k,i (cid:107) ≤ 1 , we have 0 < p k < 1 , ∀ 1 ≤ k ≤ K .
We now solve the equation ∂ ˜ L ∂ m k,i = 0 .
Assume that λ k,i = 0 , and then we have,
[Equation]
Since 1 − p k
= (cid:80) Kj (cid:54) =
k p j and Eq.
(11), multiplying ˆ w k by both sides of Eq.
(17), we have,
[Equation]
18) which contradicts with 0 < p k < 1 .
Then we have the other case λ k,i > 0 .
Based on the KKT condition, the global optimality ˆ m k,i satisﬁes that
[Equation]
The equation ∂ ˜ L ∂ ˆ m k,i = 0 leads to:
[Equation]
Based on Eq.
(11), for any j (cid:48)
(cid:54) =
k , multiplying ˆ w j (cid:48) by both sides of Eq.
(20), we have,
[Equation]
Since ∀ k ∈ [1 , K ] , p k > 0 , we have ˆ m Tk,i ˆ w j (cid:48) < 0 .
Then for any j 1 , j 2 (cid:54) =
k ,
[Equation]
The function f ( x ) = exp( x ) /x is monotonically increasing when x < 1 .
So, Eq.
(22) indicates that
[Equation]
and
[Equation] [Equation]
Multiplying ˆ w k by both sides of Eq.
(20), we have,
[Equation] [Equation]
Combing Eq.
(24) and Eq.
(25), we have,
[Equation] [Equation]
Based on p j = 1 − p k K − 1 , ∀ j (cid:54) =
k , and Eq.
(12), we can rewrite Eq.
(20) as:
[Equation] [Equation]
which means that ˆ m k,i is aligned with ˆ w k , i.e., cos ∠ ( ˆ m k,i , ˆ w k )
= 1 .
Given that (cid:107) ˆ w k (cid:107) = 1 and (cid:107) ˆ m k,i (cid:107) =
1 (Eq.
(19)), we have,
[Equation]
and Eq.
(26) leads to:
[Equation]
Therefore, for any column vector ˆ m k,i in ˆ M , we have,
[Equation]
which concludes the proof for CE loss.
[Equation]
[Table] [Table]
B A PPENDIX : I MPLEMENTATION D ETAILS
Datasets.
We conduct our experiments on three FSCIL benchmark datasets including miniImageNet (Russakovsky et al., 2015), CIFAR-100 (Krizhevsky et al., 2009)
[Table]
and CUB-200 (Wah et al., 2011).
miniImageNet is a variant of ImageNet with an image size of 84 × 84 .
It has 100 classes with each class containing 500 images for training and 100 images for testing.
CIFAR-100 has the same number of classes and images, and the image size is 32 × 32 .
CUB-200 is a dataset for ﬁne-grained image classiﬁcation containing 11,788 images of 200 classes in a resolution of 224 × 224 .
There are 5,994 images for training and 5,794 images for testing.
We follow the standard experimental settings in FSCIL (Tao et al., 2020b; Zhang et al., 2021).
For both miniImageNet and CIFAR-100, the base session ( t = 0 ) contains 60 classes, and a 5-way 5-shot (5 classes and 5 images per class) problem is adopted for each of the 8 incremental sessions ( 1 ≤ t ≤ 8 ).
For CUB-200, 100 classes are used in the base session, and there are 10 incremental sessions, each of which is 10-way 5-shot.
Architectures.
Prior studies widely adopt ResNet-12, ResNet-18, and ResNet-20 (He et al., 2016) for FSCIL experiments.
As shown in Table 5, we compare the backbone networks used in different studies.
For miniImageNet and CIFAR-100, we use ResNet-12 following Hersche et al.
For CUB-200, we use ResNet-18 (pre-trained on ImageNet) following other studies.
We adopt a two-layer MLP block as the projection layer following the practice in Peng et al.
Training Details.
We adopt the standard data pre-processing and augmentation schemes including random resizing, random ﬂipping, and color jittering (Tao et al., 2020b; Zhang et al., 2021; Peng et al., 2022).
We train all models with a batchsize of 512 in the base session, and a batchsize of 64 (containing new session data and intermediate features in the memory) in each incremental session.
On miniImageNet, we train for 500 epochs in the base session, and 100-170 iterations in each incremental session.
The initial learning rate is 0.25 for base session, and 0.025 for incremental sessions.
On CIFAR-100, we train for 200 epochs in the base session, and 50-200 iterations in each incremental session.
The initial learning rate is 0.25 for both base and incremental sessions.
On
CUB-200, we train for 80 epochs in the base session, and 105-150 iterations in each incremental session.
The initial learning rates are 0.025 and 0.05 for base session and incremental sessions, respectively.
In all experiments, we adopt a cosine annealing strategy for learning rate, and use SGD with momentum as optimizer.
Our code will be publicly available in the ﬁnal version.
C A PPENDIX : M ORE R ESULTS
Our experimental result on CUB-200 is shown in Table 4.
We achieve a better accuracy in the last session than most of the baseline methods.
Although we do not surpass ALICE in the last session on CUB-200, we still have the best average accuracy among all methods.
We also visualize the average cosine similarities between feature and classiﬁer of the same class, i.e., Avg k { cos ∠ ( m k − m G , w k ) } and the trace ratio of within-class covariance to between-class covariance, tr(Σ W ) / tr(Σ B ) .
A higher average cos ∠ ( m k − m G , w k ) indicates that feature centers are more closely aligned with their corresponding classiﬁer prototypes of the same class.
As shown in Figure 4, the values of our method are consistently higher than those of the baseline method.
Figure 4a and Figure 4d reveal that our method has a better feature-classiﬁer alignment in each session of the incremental training on both train and test sets.
When we measure on all the encountered classes by each session in
Figure 4b and Figure 4e, the metric for our method does not change obviously after the 4-th session on train set, while the metric for the baseline method keeps decreasing as training incrementally.
Especially for the base session classes, our method is able to keep the metric stable on both train and test sets after the decline of the ﬁrst 3-4 sessions, as shown in Figure 4c and Figure 4f.
As a comparison, the baseline method cannot mitigate the deterioration.
Given that the base session has the most classes, the performance on base session classes largely decides the ﬁnal accuracy in the last session for FSCIL.
Therefore, the superiority of our method can be attributed to our ability of keeping the feature-classiﬁer alignment well for base session classes.
The within-class covariance Σ W and the between-class covariance Σ B are deﬁned as:
[Equation]
and
[Equation]
where m k,i is the feature of sample i in class k , m k is the within-class mean of class k features, and m G denotes the global mean of all features.
A lower within-class variation with a higher betweenclass variation corresponds to a better Fisher Discriminant Ratio.
As shown in Figure 5, we compare the trace ratio of within-class covariance to between-class covariance between our method and the baseline method.
We observe similar patterns to Figure 4.
Concretely, the trace ratio metric of our method is consistently lower that of baseline.
For the base session classes, the metric of our method increases more mildly, which corroborates our ability of maintaining the performance on the old classes, and is in line with the indications from Figure 3 and Figure 4.
[Figure]
current session (b and e), and only the base session classes (c and f), on train set (a, b, c) and test set (d, e, f), for models trained after each session on miniImageNet.
[Figure]
where Σ W is the within-class covariance, and Σ B denotes the between-class covariance.
Statistics are performed among classes in each session (a and d), all encountered classes by the current session (b and e)