Margin-Based Few-Shot Class-Incremental Learning with Class-Level Overﬁtting Mitigation
Abstract
Few-shot class-incremental learning (FSCIL) is designed to incrementally recognize novel classes with only few training samples after the (pre-)training on base classes with sufﬁcient samples, which focuses on both base-class performance and novel-class generalization.
A well known modiﬁcation to the base-class training is to apply a margin to the base-class classiﬁcation.
However, a dilemma exists that we can hardly achieve both good base-class performance and novel-class generalization simultaneously by applying the margin during the base-class training, which is still under explored.
In this paper, we study the cause of such dilemma for FSCIL.
We ﬁrst interpret this dilemma as a class-level overﬁtting (CO) problem from the aspect of pattern learning, and then ﬁnd its cause lies in the easilysatisﬁed constraint of learning margin-based patterns.
Based on the analysis, we propose a novel margin-based FSCIL method to mitigate the CO problem by providing the pattern learning process with extra constraint from the margin-based patterns themselves.
Extensive experiments on CIFAR100, Caltech-USCD Birds200-2011 (CUB200), and mini ImageNet demonstrate that the proposed method effectively mitigates the CO problem and achieves state-of-the-art performance.
Introduction
With the development of deep learning, deep neural networks gradually demonstrate superior performance on the recognition of pre-deﬁned classes with large amount of training data [19, 10].
However, the model’s generalization capability on the downstream novel classes is much less explored and still needs to be improved [13, 9].
To deal with this problem, the few-shot class-incremental learning (FSCIL) task [11, 18, 2, 22, 28, 30] comes into sight.
FSCIL ﬁrst (pre-)trains a model on a set of pre-deﬁned classes (base classes), and then generalizes the model to the incremental novel classes with only few training samples, simulating human’s ability of continually learning novel concepts with only few examples, and emphasizing both the performance on the pre-deﬁned base classes and the generalization on the downstream novel classes.
However, a dilemma is recently revealed [13, 5, 6, 7] that better loss functions, which lead to higher performance on the pre-training data, could lead to worse generalization on the downstream tasks.
As introduced by [16] and depicted in Fig.
1, similar phenomenon also exists in the FSCIL task that a positive classiﬁcation margin [16, 25, 8, 21] applied to the classiﬁcation of the base-class (pre-)training could lead to higher base-class performance but lower novel-class performance, while a negative margin could result in lower base-class performance but increase the novel-class performance.
Although this dilemma widely exists in the tasks involving novel-class generalization such as few-shot learning (FSL) and FSCIL, only few works [16] tried to explore its cause, and can hardly be used to handle it.
Due to space limitation, we will provide extended related works in the appendix.
No Margin
Positive Margin
Negative Margin
Ours
B a s e
C l a ss
N ov e l C l a ss
In this paper, we study the cause of the dilemma in the margin-based classiﬁcation for the FSCIL problem from the the aspect of pattern learning.
We ﬁnd this dilemma can be understood as a classlevel overﬁtting (CO) problem, which can be interpreted by the ﬁtness of the learned patterns to each base class.
The ﬁtness determines how much the learned patterns are speciﬁc to some base classes or shared among classes, making the learned patterns either discriminative (tend to overﬁt base classes) or transferable (tend to underﬁt base classes) and causes the dilemma.
Based on the interpretation, we discover the cause of the dilemma lies in the easily-satisﬁed constraint of learning shared or class-speciﬁc patterns.
Therefore, we further design a novel FSCIL method to mitigate the dilemma of CO by providing the pattern learning process with extra constraint from margin-based patterns themselves, improving performance on both base and novel classes as shown in Fig.
1, and achieving state-of-the-art performance in terms of the all-class accuracy.
Our contributions are:
• We interpret the dilemma of the margin-based classiﬁcation as a class-level overﬁtting problem from the aspect of pattern learning.
We ﬁnd the cause of the class-level overﬁtting problem lies in the easily-satisﬁed constraint of learning shared or class-speciﬁc patterns.
• We propose a novel FSCIL method to mitigate the class-level overﬁtting problem based on the interpretation and analysis of the cause.
• Extensive experiments on three public datasets verify the rationale of the model design, and show that we can achieve state-of-the-art performance.
Interpreting the Dilemma of Few-Shot Class-Incremental Learning
In this section, we ﬁrst describe the Few-Shot Class-Incremental Learning (FSCIL) task and the baseline model, and then conduct experiments to analyze the dilemma.
2.1 Task and Baseline Description
The FSCIL task aims to incrementally recognize novel classes with only few training samples.
Basically, the model is ﬁrst (pre-)trained on a set of base classes with sufﬁcient training samples (a.k.a. base session), then confronted with novel classes with limited training samples (a.k.a. incremental session), and ﬁnally required to recognize test samples from all encountered classes.
Speciﬁcally, given the base session dataset D
= { ( x i , y i ) } n 0 i =1 with the label space Y 0 , the model is trained to recognize all | Y 0 | classes from Y 0 by minimizing the loss ∑
[Equation]
where L ( · , · ) is typically a cross-entropy loss, ϕ ( · ) is the predictor which is composed of a backbone network f ( · ) for feature extraction and a linear classiﬁer, represented as ϕ ( x )
= W ⊤ f
( x ) where ϕ ( x ) ∈ R N 0 × 1 , W ∈ R d × N 0 and f ( x ) ∈ R
d × 1 .
Typically, f ( x ) and the W are L 2 normalized [28].
When the k th incremental session comes, the model needs to learn from its training data D k
= { ( x i , y i ) } n k i =1 .
The weight of the classiﬁer will be extended to represent the novel label space Y k imported by this session, represented as W = { w 01 , w 02 , ..., w 0 | Y 0 | } ∪ ... ∪ { w k 1 , ..., w k | Y k | } where w kj denotes the weight of the classiﬁer corresponding to the j th class of the k th session.
A strong baseline [28] is to freeze model’s parameters to avoid the catastrophic forgetting brought by the ﬁnetuning on novel-classes.
For the incremental sessions (i.e., k > 0), the average of the features extracted from the training data will be used as the classiﬁer’s weight [28] (a.k.a. prototype) as w kj =
[Equation]
i = 1 f ( x i ) , where n jk denotes the number of training samples in the class j for the session k .
As this baseline focuses on the base-class training, in this paper, the term training , if not otherwise stated, refers to the base-class training.
Finally, the performance of the k th session will be obtained by classifying the test samples from all ∑ ki =
| Y i | encountered classes.
2.2 Margin-Based Classiﬁcation
A well known modiﬁcation to base-class training loss (Eq. 1) is to integrate a margin [16, 8, 25] as [Equation] x i ,
[Equation]
where w y i refers to the classiﬁer weight for class y i , τ is typically set to 16.0 and m is the margin.
As analyzed in [16], empirically a dilemma exists that a positive margin could improve the baseclass performance but harm the novel-class generalization, and reversely, a negative margin could contribute to the novel-class performance but decrease that of the base classes as shown in Fig.
Similar phenomenon has been observed in other works such as [13] that a better loss function for the pre-training task could harm the generalization on downstream tasks.
2.3 Interpretation of Class-Level Overﬁtting from Pattern Learning View
Experiments are conducted on the CIFAR100 [15] dataset and reported in Fig.
CIFAR100 contains 100 classes in all.
As split by [22], 60 classes are chosen as base classes, and the remaining 40 classes (with 5 training samples in each class) are chosen as novel classes 2 .
Experiments are conducted on the last incremental session, where all 100 classes are involved.
From Fig.
2 (left), we can see that as the margin increases, the base-class accuracy increases while the novel-class accuracy decreases, which is consistent with [16] and validates the dilemma exists.
Compared with the well-known overﬁtting between the training and testing data, such dilemma, although all validated on testing data, is more like the overﬁtting to base classes instead of samples.
Therefore, we term it as class-level overﬁtting (CO) .
Additionally, the balance is reached when no margin is added, i.e., FSCIL cannot be improved by simply applying the margin.
For such dilemma, [16] gave an explanation by the degraded mapping from novel to base classes.
However, it could hardly be used to develop methods for handling such dilemma.
In this section, we go a step further to explain this phenomenon from the aspect of pattern learning for developing methods to handle it.
A pattern denotes a part of information that the model extracts from the input, which is a ﬁnergrained level of analyzing the model’s behavior.
As studied in the interpretability of deep nets [29, 1], each channel in the feature extracted by deep networks could correspond to a certain pattern of the input 2 , which can be viewed as to compose the base and novel classes [31].
Therefore, we conduct experiments on feature channels to study the patterns learned by applying different margins.
2.3.1 Class-Level Overﬁtting Interpreted by Pattern Fitness to Base Classes
Pattern’s ﬁtness to each base class.
We ﬁrst evaluate the sparsity of the base-class patterns, which is measured by the L 1 norm of each feature vector.
As the extracted features are L 2 normalized, the smaller the L 1 norm is, for each feature, the sparser the patterns with high activation are.
Results are plotted in Fig.
2 (mid), where we can see a consistent decrease in L 1 when the margin increases, which means the model needs less activated patterns (channels) to represent each base class.
As the number of activated patterns decreases, the effectiveness of each activated pattern must increase to account for the performance increase in the base-class pre-training in Fig.
2 (left).
Therefore, we hold that as the margin increases, the patterns learned by the model could ﬁt each base class better.
Pattern ﬁtness measured by the template-matching score.
To further verify the ﬁtness increase, we view each pattern as a semantic template and measure its matching score to each base class.
As analyzed in [29] and [1], each pattern can be understood as a template [3] for the model to match the input (so that each class would has its own set of templates for recognition), and the activation can be viewed as the matching score.
Therefore, we could know how much all patterns ﬁt (match) each class by ﬁnding the most important patterns for each class and compare their activation.
As analyzed in [29] and [31], patterns (channels) with higher weights in the classiﬁcation layer are more important, and the most important ones dominate the model decisions.
Therefore, given an input, we select its most important patterns by the top classiﬁcation weights of its ground-truth class, and record the average activation on these patterns.
The Mean value of such Top Activation across all samples is denoted as MTA in Fig.
2 (mid).
As can be seen, as the margin increases, MTA increases consistently, which further veriﬁes patterns’ increase in ﬁtting each base class.
Better pattern ﬁtness, worse pattern transferability.
As each pattern could ﬁt a corresponding base class better, its discriminability increases accordingly, but could it be transferred across classes?
To answer it, we test the transferability of patterns.
Since classes are related (e.g., cat and tiger), transferable patterns activated in one class could also be activated in other classes (e.g., felid patterns).
Therefore, we ﬁrst ﬁnd important patterns for each base class by the classiﬁcation weights, then record activation of these patterns on other classes, and measure the transferability of patterns by the mean value of such other-class-activation.
The results are plotted in Fig.
2 (right).
As can be seen, the transferability consistently decreases when the margin increases.
Combine this result with
Fig.
2 (mid), we hold that patterns tend to be less transferable when they ﬁt each base class better.
Discussion.
The ﬁtness also reﬂects the how much the given pattern is speciﬁc to a base class.
Imagine the extreme situation where each base-class only needs one pattern for representation, the ﬁtness would reach its upper bound to make such pattern thoroughly speciﬁc to the corresponding class.
Therefore, we interpret that the higher the margin is, the more speciﬁc (overﬁtting) the patterns are to each base class, which makes patterns more discriminative but less transferable.
Meanwhile, the lower the margin is, the more the patterns could be shared between classes (underﬁtting), making patterns more transferable but less discriminative.
The CO dilemma lies in that patterns can hardly be both class-speciﬁc and shared among classes by simply applying the classiﬁcation margin.
2.3.2 Inherent Class Relations Lead to the Change in Pattern’s Base-Class Fitness
Pattern’s ﬁtness negatively inﬂuences class relations.
In Fig.
2 (right), we also plot the class relations w.r.t.
the margins.
The class relations are measured by the average of cosine similarities between every two classes’ prototypes.
As can be seen, the relation drops as the margin grows, in consistent with the trend of the patterns’ transferability.
This is rationale because the if two prototypes share some patterns, the activation of the corresponding channels will be similar, making the cosine similarity larger.
As the transferability of patterns is negatively related to pattern’s baseclass ﬁtness, we hold that the class relations are also negatively related to the base-class ﬁtness.
Inherent class relations inﬂuence pattern’s ﬁtness.
The margin applied to the classiﬁcation directly modiﬁes the decision boundary between every two classes, and the decision boundary is related to the relationship between every two classes.
Therefore, we study how the class relation inﬂuences the pattern’s ﬁtness to base classes.
Speciﬁcally, given 60 base classes, for the model trained without margins, we ﬁrst calculate the cosine similarity between every two different classes, which gives 60 × (60 - 1) / 2 = 1,750 relations denoted as R 0 , which represents the inherent relations between all classes.
Similarly, we calculate 1,750 relations for the model trained with positive and negative margins respectively, denoted as R pos and R neg .
Then we calculate D pos
=
R pos − R and D neg =
R neg − R 0 .
Finally, we sort R 0 from small to large, and use the sorted index to arrange D neg and D pos .
As plotted in Fig.
3, the blue dots denote D neg (left) or D pos
(right), and the blue curve are dots ﬁtted by multinomial functions.
We can see that the change in class relations is positively related to the inherent class relations for the negative margins, while is negatively related to the inherent class relations for the positive margins, especially for index larger than 1,000.
Since class relations are negatively related to pattern’s ﬁtness to base classes, the more the class relation increases, the less the patterns could ﬁt the corresponding base class.
Therefore, the results can be understood that the more inherently similar two classes are, by applying a negative margin, the less the patterns could ﬁt the given base class, i.e., the more the patterns are shared by given classes, making these classes’ representations more similar; by applying a positive margin, the more the patterns could ﬁt the given base classes, making these classes’ representations more dissimilar.
Conclusion and Discussion.
Therefore, as shown in Fig. 4, we interpret the pattern learning process as follows.
Given a negative margin, the decision boundary between two classes are confused, making more samples fall into the overlapping region between two classes.
This makes it possible to learn patterns shared by these two classes and hinder the pattern from ﬁtting the given base class, and makes patterns more transferable but less discriminative.
The more similar inherently two classes are (i.e., larger overlapping region in Fig. 4), the more shared patterns can be learned (e.g., more patterns could be shared between cats and tigers than cats and air-planes), therefore making these classes’ representations more similar.
On the contrary, given a positive margin, the decision boundary between two classes should be well separated, pushing the model to learn patterns ﬁtting (i.e., speciﬁc to) each class, which are more discriminative but less transferable.
The more similar inherently two classes are, the harder the learning is and the larger the training loss will be, therefore making the patterns ﬁt each class more and making these classes’ representations more dissimilar.
Mitigating the Dilemma of Few-Shot Class-Incremental Learning
In this section, we ﬁrst analyze the cause of the dilemma in margin-based classiﬁcation based on the above interpretation, then we propose our method (named as Class-Level Overﬁtting Mitigation, CLOM) to mitigate the CO dilemma based on the analysis, as shown in Fig.
3.1 Analysis on the Cause of Class-Level Overﬁtting
Based on the above analysis, we can ﬁnd the learning of negative-margin-based patterns loosely constrains the given pattern to be shared by both classes.
However, such constraint is easily satisﬁed by ineffective patterns as simple as edges or corners, which could lead to the low discriminability of negative-margin-based patterns.
Similarly, the learning of positive-margin-based patterns loosely constrains the given pattern to be speciﬁc to the given class.
However, such constraint could be satisﬁed by easily ﬁnding patterns sharing no information with other classes, such as ﬁnding some complex texture only speciﬁc to the given class, which could lead to the low transferability of positivemargin-based patterns.
Such easy-constraint problems push patterns to be only class-speciﬁc or shared among classes, making pattern effective in one scenario ineffective in other ones.
To verify the above claims, we analyze how simple or complex the learned patterns are.
Inspired by [14], we quantitatively measure the simplicity/complexity of patterns by the similarity between the extracted feature and the simplest feature (e.g., corner or edge features).
We ﬁrst use the baseline model to train on CIFAR100, and use the ﬁrst convolutional layer as the simplest feature extractor (denoted as f simple ), since many works (e.g., [27]) has shown that the ﬁrst convolutional layer tends to capture corners or edges.
Then, we train models with different margins, and use the backbone network for feature extraction (denoted as f target ).
After that, we extract f simple and f target features from all images in base classes.
Finally, we compare the CKA similarity [14] for measuring the similarity between f simple and the f target . For a sanity check, we ﬁrst report the similarity between different layers within the baseline model. [Table] We can see that the shallower the layer is, the higher the CKA similarity would be, which means the more similar they are to the f simple , i.e. the patterns are simpler, more transferable but less discriminative [26].
Then, we report the comparison of the CKA similarity between f simple and f target (backbone feature) of the baseline model trained with margins.
[Table]
We can see that by applying a negative margin, the CKA similarity clearly increases (even larger than that of the second last convolutional layer when margin < -0.3), showing that the backbone network captures patterns more similar to simplest ones such as edges or corners, which veriﬁes that the model tends to learn simple patterns that are easily shared between classes.
When applying a positive margin, the captured patterns grow to be more complex and tend to overﬁt base classes, making the CKA much smaller than baseline model’s backbone-output, which veriﬁes that the model tends to learn complex patterns that are easily speciﬁc to a given base class.
Therefore, the key to mitigating the CO dilemma is to extra constrain the pattern learning process.
3.2 Mitigating Class-Level Overﬁtting by Providing Extra Constraint
Negative-Margin-based (NM) patterns are more transferable and class-shared, while PositiveMargin-based (PM) patterns are discriminative and class-speciﬁc.
These characteristics are similar to the behavior of features from the shallow and deep layers of deep networks [26, 32].
Generally, shallow-layer features encode low-level patterns that are easily shared by most objects therefore more transferable, such as edge or corner, while the deep-layer features encode high-level patterns that are semantically related to only few object classes therefore more discriminative.
Such similarity inspires us to view PM patterns as high-level patterns while view NM patterns as (relatively) low-level patterns, and build PM patterns from the NM patterns, just like building high-level features from low-level features in deep networks.
As the learning of NM patterns is inﬂuenced by PM patterns, this design could beneﬁt the learning of NM patterns by providing extra constraint from the learning of PM patterns, and vice versa, which could therefore handle the easy-constraint problem.
Speciﬁcally, given an feature extractor f ( · ) , we add one more layer to construct the PM feature as
[Equation]
) where F ( · ) is the PM feature extractor, g ( · ) is typically composed of a fully-connected layer, a batchnormalization layer [12] and an activation layer.
During training, two classiﬁcation loss is applied to both F ( · ) and f ( · ) with positive and negative margins respectively, represented as
L ( x i , y i ) =
L neg ( x i , y i )
+ L pos ( x i , y i ) (4) =
− log e τ ( w yi f ( x i ) − m neg ) e τ ( w yi f ( x i ) − m neg ) + ∑ j ̸ = y i e τw j f ( x i ) − λ · log e τ ( w Pyi F ( x i ) − m pos )
e τ ( w Pyi F ( x i ) − m pos )
+ ∑
j ̸ = y i e τw Pj F ( x i ) , (5) where w Pj ∈ R 1 ×
d P denotes the classiﬁer weights for F ( · ) corresponding to class j , m neg denotes the negative margin for f ( · ) , m pos is the positive margin for F ( · ) , and λ is typically set to 1.0.
Given this modiﬁcation, the NM patterns f ( · ) are utilized to construct the discriminative
PM patterns F ( · ) .
Since g ( · ) is relatively too simple to capture complex patterns, f ( · ) will be pushed to be discriminative (validated in Fig.
7), instead of casually learning ineffective patterns shared between similar classes.
Similarly, as F ( · ) are built from the transferable f ( · ) by the simple g ( · ) , the transferability could be maintained by pushing PM patterns to ﬁt the corresponding base class by transferable information, which improves the transferability of F ( · ) (validated in Fig.
During testing, given the improved f ( · ) and F ( · ) , the ﬁnal feature would be their concatenation.
3.3 Further Mitigation of Class-Level Overﬁtting by Integrating Class Relations
Based on the architecture design, we propose to further mitigate the CO problem through boosting the discriminability and tranferability of NM and PM patterns by integrating the class relations.
For easy understanding, we ﬁrst introduce the modiﬁcation to the classiﬁcation for f ( · ) , and later introduce that for F ( · ) .
To modify the margin-based classiﬁcation from a single margin to margins related to class relations, we ﬁrst move the margin from the ground-truth logit to all other logits as
[Equation]
where A is the adjacency matrix between all classes, m ( · ) maps the adjacency value between two classes to a margin.
Given this modiﬁcation, if the margin is set to the original ﬁxed value as Eq. 2, the decision boundary between class y i and other classes remains the same, therefore the new loss in Eq.
6 could be an approximation of the original loss in Eq. 2.
For measuring class relations, we choose to utilize the adjacency matrix of all classes.
As both the feature and the classiﬁer’s weights are L 2 normalized, the adjacency can be measured by the cosine similarity between every two class prototypes as A ij = cos ( P i , P j ) , where P i is the prototype for class i , which is typically set to w i of the classiﬁer.
If two classes are identical, the cosine similarity would reach its upper bound to 1.0.
Meanwhile, the average of class relations reﬂects a global margin that is effective for most classes.
Therefore, we design to interpolate the margin from a global effective value to a pre-deﬁned upper bound value as
[Equation]
[Equation] where m upper is a hyper-parameter controlling the margin for the upper-bound class relation (i.e., two identical classes), m ave is another hyper-parameter controlling the margin for the average class relations, set to the same value as that in Eq. 2, and A ave is the average class relations calculated as A ave
[Equation]
For F ( · ) , we adopt the same modiﬁcation as f ( · ) , which replaces f ( · ) with F ( · ) in Eq. 6, and replaces A with A P in Eq.
6 where A Pij = cos ( w Pi , w Pj ) .
Therefore, the ﬁnal training objective is
L ( x i , y i )
= − log
[Equation]
where m p ( · ) and m n ( · ) are two interpolate functions as Eq.
7 with positive and negative hyperparameters ( m Pupper , m Pave ) and ( m upper , m ave ) respectively.
In experiments (Fig. 9), we ﬁnd it beneﬁcial to have m upper < m ave and m P upper > m P ave , which means for classes with higher similarities, the relation mapping module enables the model to learn more shared (transferable) patterns by applying smaller negative margins, and learn more classspeciﬁc (discriminative) patterns by applying larger positive margins.
Moreover, due to the connection between NM and PM patterns, the improved transferability in NM patterns would be transmitted to PM patterns and vice versa, as validated in Fig.
9, which further mitigates the CO dilemma.
[Table] [Table] After the base-session training, the feature extractor F ( · ) and f ( · ) will be applied to the training data of each session to obtain the extended classiﬁer weight W = { w 01 , w 02 , ..., w 0 | Y 0 | } ∪ ... ∪ { w k 1 , ..., w k | Y k | } by averaging extracted features, and the ﬁnal performance of each session will be obtained based on the classiﬁcation of all the encountered classes’ test samples.
Experiments
In this section, we ﬁrst introduce the experiment settings, then compare the proposed method with the state-of-the-art methods, and ﬁnally report the ablation study for the effectiveness of each design.
4.1 Datasets
Datasets include CIFAR100 [15], Caltech-UCSD Birds-200-2011 (CUB200) [24] and miniImageNet [23] as listed in Tab. 3 following the split in [22].
For details, please refer to the appendix.
4.2 Implementation Details
The implementation is based on CEC’s code [28], and our code will be released 3 .
For CIFAR100, we set d P =256, set m ave =
-0.2, set m upper =
-0.5, and we have m Pave =0.1 and m Pupper =0.2.
For CUB200, we scale the learning rate of the backbone network to 10% of the global learning rate since the pretraining of the backbone is adopted [30, 28], and set d P to 8192.
Then we have m ave =
-0.2 and m upper =
-0.25 and m Pave =0.3 and m Pupper =0.6.
For mini ImageNet, we set d P to 4096, and have m ave =
-0.2 and m upper =
-0.5 and m Pave =0.1 and m Pupper =0.2.
Please refer to appendix for details.
4.3 Comparison with the State-of-the-Art
Comparisons with the state-of-the-art works are listed in Tab. 4 and Fig.
6, where we can achieve state-of-the-art performance on all three datasets.
Speciﬁcally, we can ﬁrst see that our method, as a prototype-based method (e.g., CEC [28]), could signiﬁcantly outperform ﬁnetune-based methods (e.g., iCaRL [18]).
This is because the few-shot training data could not provide sufﬁcient information for novel-class learning, therefore directly freezing parameters on novel classes could reduce the catastrophic-forgetting problem brought by the ﬁnetuning.
Our method also outperforms other prototype-based methods, this is because the core of the prototype-based methods lies in the metric learning [20].
As empirically proved by current works [21, 25, 8, 16], applying the marginbased classiﬁcation could effectively improve the embedding space learned by metric-based methods.
Since our method handles the difﬁculty of applying margin-based classiﬁcation (i.e., class-level overﬁtting) to the FSCIL task, our method could beat these prototype-based ones.
For detailed numbers of CIFAR100 and mini ImageNet, please refer to the appendix.
4.4 Ablation Study of Modules
The ablation study of each module is reported in Tab. 5, where g ( · ) denotes adding another layer,
Margin denotes the applicant of margin-based classiﬁcation, and Relation refers to the relation map- [Table] ping of the margin.
We study from three aspects: Base -class, Novel -class [Table] and Overall accuracy of the last incremental session.
From Tab. 5, we can see that
• Simply applying another layer cannot consistently improve the performance .
• The designed architecture could mitigate class-level overﬁtting .
Compared with experiments in Fig.
2 where no Overall performance improvements can be obtained by simply adding margins, the performance here is clearly improved by adding margins on the designed architecture.
Moreover, the overall improvements originate from not only the improved Base performance, but also the boosted Novel performance, demonstrating the mitigation of class-level overﬁtting (CO) problem.
• Relation mapping could further improve performance by mitigating class-level overﬁtting .
4.5 Veriﬁcation of Class-Level Overﬁtting Mitigation
Mitigating the class-level overﬁtting by the architecture design.
We ﬁrst compare our method with the baseline method which applies the margin directly to the backbone feature in Fig.
7 and 8, so as to validate the mitigation of CO.
In Fig.
7, we apply the negative margin to the backbone (NM) feature while applying no margin to the PM feature, and see the accuracy of the NM feature against the baseline.
As can be seen, while keeping the improvements on novel classes (mid), our method (orange) maintains the base-class accuracy (right), in contrast to a sharp decrease of the baseline (blue) when the margin decreases, which makes our NM feature signiﬁcantly outperforms the baseline in terms of the overall accuracy (left).
Similarly, in Fig.
8, our PM feature also maintains a higher performance on novel classes when the margin increases.
These experiments validate the mitigation of the CO dilemma: by implicitly guiding the learning of the PM and NM features by each other, the transferability or discriminability drop on them can be mitigated respectively.
Further mitigating class-level overﬁtting by relation mapping.
To validate the relation mapping module, in Fig.
9 we ﬁxed the average margin (i.e., m ave =-0.2 in Fig.
9.1st and m Pave =0.3 in
Fig.
9.2nd) and experiment with different upper margins (i.e., m upper in Fig.
9.1st and m Pupper in
Fig.
9.2nd).
Similar to Fig.
7, when conducting experiments on one branch, the hyper-parameters of the other branch are ﬁxed.
We can see the relation mapping module can indeed help the learning, as the upper margins are signiﬁcantly different from the average margin.
We also plot the NM and PM performance on base classes with negative margins in Fig.
9(3rd), and those on novel classes with positive margins in Fig.
9(4th).
We can see the NM and PM performance are improved simultaneously, which validates the improvements brought by the extra guidance of each features.
Therefore, the relation mapping module can also help to mitigate the class-level overﬁtting problem by building
PM features with more transferable patterns, and vice versa.
Moreover, it is interesting to ﬁnd the performance is improved when m upper < m ave and m Pupper > m Pave , which means for classes with higher similarities, the relation mapping module enables the model to learn more shared patterns by applying negative margins with larger absolute value, and learn more class-speciﬁc patterns by applying larger positive margins.
Also, this is consistent with the results in Fig.
4.6 Mitigating the Easy-Constraint Problem
Following experiments in Tab. 1, we also utilize the CKA similarity between the simplest feature f simple and the NM/PM feature to validate the mitigation of the easy-constraint problem. By applying our method, for the NM (or PM) patterns, we set f target to f ( · ) (or F ( · ) ) in Fig.5 and set the positive/negative margin to 0.3/-0.4, and report the CKA below. [Table]
In Tab. 6 (left), compared with Tab. 2, CKA decreases clearly compared with the baseline method, which validates that NM patterns learned by our method are more complex and less similar to edges and corners than the baseline method, verifying the mitigation of the easily-satisﬁed constraint problem by extra supervision from the learning of PM patterns. Also, in Tab. 6 (right), compared with
Tab. 2, CKA is much larger than that of the baseline method, which validates that the PM patterns are more similar to the simplest patterns than the baseline method, which makes it less overﬁtting the base classes, verifying the mitigation by extra supervision from the learning of NM patterns.
Conclusion
In this paper, we focus on the dilemma in the margin-based classiﬁcation for FSCIL.
We ﬁrst interpret the dilemma as a class-level overﬁtting problem from the aspect of pattern learning, then ﬁnd the cause of this problem lies in the easily-satisﬁed constraint of learning shared or class-speciﬁc patterns.
Based on the analysis, we design a method (CLOM) to mitigate the dilemma by constructing
PM patterns from NM patterns, and mapping class relations into class-speciﬁc patterns.
Extensive experiments on three public datasets validate the effectiveness and outstanding performance of the proposed method.
Acknowledgements
This work is supported by National Natural Science Foundation of China under grants U1836204,
U1936108, 62206102, Science and Technology Support Program of Hubei Province under grant 2022BAA046, and 2022 CCF-DiDi Gaiya Young Scholar Research Fund.