Margin-Based Few-Shot Class-Incremental Learning with Class-Level Overﬁtting Mitigation
Abstract
Few-shot class-incremental learning (FSCIL) is designed to incrementally recognize novel classes with only few training samples after the (pre-)training on baseclasses with sufﬁcient samples, which focuses on both base-class performance andnovel-class generalization.
A well known modiﬁcation to the base-class trainingis to apply a margin to the base-class classiﬁcation.
However, a dilemma existsthat we can hardly achieve both good base-class performance and novel-class generalization simultaneously by applying the margin during the base-class training,which is still under explored.
In this paper, we study the cause of such dilemma for
FSCIL.
We ﬁrst interpret this dilemma as a class-level overﬁtting (CO) problemfrom the aspect of pattern learning, and then ﬁnd its cause lies in the easily-satisﬁedconstraint of learning margin-based patterns.
Based on the analysis, we proposea novel margin-based FSCIL method to mitigate the CO problem by providingthe pattern learning process with extra constraint from the margin-based patternsthemselves.
Extensive experiments on CIFAR100, Caltech-USCD Birds-200-2011(CUB200), and mini ImageNet demonstrate that the proposed method effectivelymitigates the CO problem and achieves state-of-the-art performance.
Introduction
With the development of deep learning, deep neural networks gradually demonstrate superior performance on the recognition of pre-deﬁned classes with large amount of training data [18, 10].
However,the model’s generalization capability on the downstream novel classes is much less explored and stillneeds to be improved [13, 9].
To deal with this problem, the few-shot class-incremental learning(FSCIL) task [11, 17, 2, 21, 26, 28] comes into sight.
FSCIL ﬁrst (pre-)trains a model on a set ofpre-deﬁned classes (base classes), and then generalizes the model to the incremental novel classeswith only few training samples, simulating human’s ability of continually learning novel conceptswith only few examples, and emphasizing both the performance on the pre-deﬁned base classes andthe generalization on the downstream novel classes.
However, a dilemma is recently revealed [13, 5, 6, 7] that better loss functions, which lead to higherperformance on the pre-training data, could lead to worse generalization on the downstream tasks.
As introduced by [15] and depicted in Fig.
1, similar phenomenon also exists in the FSCIL taskthat a positive classiﬁcation margin [15, 24, 8, 20] applied to the classiﬁcation of the base-class(pre-)training could lead to higher base-class performance but lower novel-class performance, while anegative margin could result in lower base-class performance but increase the novel-class performance.
Although this dilemma widely exists in the tasks involving novel-class generalization such as few-shotlearning (FSL) and FSCIL, only few works [15] tried to explore its cause, and can hardly be used tohandle it.
Due to space limitation, we will provide extended related works in the appendix.
No Margin
Positive Margin
Negative Margin
Ours
[Figure]
C
[Figure]
ss
N ov e l C l a ss
In this paper, we study the cause of the dilemma in the margin-based classiﬁcation for the FSCILproblem from the the aspect of pattern learning.
We ﬁnd this dilemma can be understood as a classlevel overﬁtting (CO) problem, which can be interpreted by the ﬁtness of the learned patterns to eachbase class.
The ﬁtness determines how much the learned patterns are speciﬁc to some base classes orshared among classes, making the learned patterns either discriminative (tend to overﬁt base classes)or transferable (tend to underﬁt base classes) and causes the dilemma.
Based on the interpretation,we discover the cause of the dilemma lies in the easily-satisﬁed constraint of learning shared orclass-speciﬁc patterns.
Therefore, we further design a novel FSCIL method to mitigate the dilemmaof CO by providing the pattern learning process with extra constraint from margin-based patternsthemselves, which achieves improved performance on both base and novel classes as shown in Fig.
1,and achieves state-of-the-art performance in terms of the all-class accuracy.
Our contributions are:
• We interpret the dilemma of the margin-based classiﬁcation as a class-level overﬁtting problem from the aspect of pattern learning.
• We ﬁnd the cause of the class-level overﬁtting problem lies in the easily-satisﬁed constraint of learning shared or class-speciﬁc patterns.
• We propose a novel FSCIL method to mitigate the class-level overﬁtting problem based on theinterpretation and analysis of the cause.
• Extensive experiments on three public datasets verify the rationale of the model design, and showthat we can achieve state-of-the-art performance.
Interpreting the Dilemma of Few-Shot Class-Incremental Learning
In this section, we ﬁrst describe the Few-Shot Class-Incremental Learning (FSCIL) task and thebaseline model, and then conduct experiments to analyze the dilemma.
2.1 Task and Baseline Description
The FSCIL task aims to incrementally recognize novel classes with only few training samples.
Basically, the model is ﬁrst (pre-)trained on a set of base classes with sufﬁcient training samples (a.k.a.base session), then confronted with novel classes with limited training samples (a.k.a. incrementalsession), and ﬁnally required to recognize test samples from all encountered classes.
Speciﬁcally, given the base session dataset D
= { ( x i , y i ) } n 0 i =1 with the label space Y 0 , the model istrained to recognize all | Y 0 | classes from Y 0 by minimizing the loss
[Equation]
where L ( · , · ) is typically a cross-entropy loss, φ ( · ) is the predictor which is composed of a backbonenetwork f ( · ) for feature extraction and a linear classiﬁer, represented as φ ( x )
= W (cid:62) f ( x ) whereφ ( x ) ∈ R N 0 × 1 , W ∈ R d × N 0 and f
( x ) ∈ R d × 1 .
Typically, f ( x ) and the W are L 2 normalized [26]. [Equation]
When the k th incremental session comes, the model needs to learn from its training data D k
={ ( x i , y i ) } n k i =1 .
The weight of the classiﬁer will be extended to represent the novel label space
Y kimported by this session, represented as W = { w 01 , w 02 , ..., w 0 | Y 0 | } ∪ ... ∪ { w k 1 , ..., w k | Y k | } where w kjdenotes the weight of the classiﬁer corresponding to the j th class of the k th session.
A strong baseline [26] is to freeze model’s parameters to avoid the catastrophic forgetting broughtby the ﬁnetuning on novel-classes.
For the incremental sessions (i.e., k > 0), the average of thefeatures extracted from the training data will be used as the classiﬁer’s weight [26] (a.k.a.
prototype)as
w kj = 1 n jk (cid:80) n jk i =1 f ( x i ) , where n jk denotes the number of training samples in the class j for thesession k .
As this baseline focuses on the base-class training, in this paper, the term training , if nototherwise stated, refers to the base-class training.
Finally, the performance of the k th session will beobtained by classifying the test samples from all (cid:80) ki =
| Y i | encountered classes.
2.2 Margin-Based Classiﬁcation
A well known modiﬁcation to base-class training loss (Eq. 1) is to integrate a margin [15, 8, 24] as
[Equation]
where w y i refers to the classiﬁer weight for class y i , τ is typically set to 16.0 and m is the margin.
As analyzed in [15], empirically a dilemma exists that a positive margin could improve the baseclass performance but harm the novel-class generalization, and reversely, a negative margin couldcontribute to the novel-class performance but decrease that of the base classes as shown in Fig.
Similar phenomenon has been observed in other works such as [13] that a better loss function for thepre-training task could harm the generalization on downstream tasks.
2.3 Interpretation of Class-Level Overﬁtting from Pattern Learning View
Experiments are conducted on the CIFAR100 [14] dataset and reported in Fig.
2. CIFAR100 contains100 classes in all.
As split by [21], 60 classes are chosen as base classes, and the remaining 40 classes(with 5 training samples in each class) are chosen as novel classes 1 .
Experiments are conducted onthe last incremental session, where all 100 classes are involved.
From Fig.
2 (left), we can see that as the margin increases, the base-class accuracy increases whilethe novel-class accuracy decreases, which is consistent with [15] and validates the dilemma exists.
Compared with the well-known overﬁtting between the training and testing data, such dilemma,although all validated on testing data, is more like the overﬁtting to base classes instead of samples.
Therefore, we term it as class-level overﬁtting (CO) .
Additionally, the balance is reached when no margin is added, i.e., FSCIL cannot be improved by simply applying the margin.
For such dilemma,[15] gave an explanation by the degraded mapping from novel to base classes.
However, it could hardly be used to develop methods for handling such dilemma.
In this section, we go a step further toexplain this phenomenon from the aspect of pattern learning for developing methods to handle it.
A pattern denotes a part of information that the model extracts from the input, which is a ﬁner-grainedlevel of analyzing the model’s behavior.
As studied in the interpretability of deep networks [27, 1],each channel in the feature extracted by deep networks could correspond to a certain pattern of theinput 1 , which can be viewed as to compose the base and novel classes [29].
Therefore, we conductexperiments on feature channels to study the patterns learned by applying different margins.
2.3.1 Class-Level Overﬁtting Interpreted by Pattern Fitness to Base
Classes
Pattern’s ﬁtness to each base class.
We ﬁrst evaluate the sparsity of the base-class patterns, whichis measured by the L 1 norm of each feature vector.
As the extracted features are L 2 normalized, thesmaller the L 1 norm is, for each feature, the sparser the patterns with high activation are.
Resultsare plotted in Fig.
2 (mid), where we can see a consistent decrease in L 1 when the margin increases,which means the model needs less activated patterns (channels) to represent each base class.
As thenumber of activated patterns decreases, the effectiveness of each activated pattern must increase toaccount for the performance increase in the base-class pre-training in Fig.
2 (left).
Therefore, wehold that as the margin increases, the patterns learned by the model could ﬁt each base class better.
Pattern ﬁtness measured by the template-matching score.
To further verify the ﬁtness increase,we view each pattern as a semantic template and measure its matching score to each base class.
Asanalyzed in [27] and [1], each pattern can be understood as a template [3] for the model to match theinput (so that each class would has its own set of templates for recognition), and the activation can beviewed as the matching score.
Therefore, we could know how much all patterns ﬁt (match) each classby ﬁnding the most important patterns for each class and compare their activation.
As analyzed in[27] and [29], patterns (channels) with higher weights in the classiﬁcation layer are more important,and the most important ones dominate the model decisions.
Therefore, given an input, we select itsmost important patterns by the top classiﬁcation weights of its ground-truth class, and record theaverage activation on these patterns.
The Mean value of such Top Activation across all samples isdenoted as MTA in Fig.
2 (mid).
As can be seen, as the margin increases, MTA increases consistently,which further veriﬁes patterns’ increase in ﬁtting each base class brought by the margin increase.
Better pattern ﬁtness, worse pattern transferability.
As each pattern could ﬁt a correspondingbase class better, its discriminability increases accordingly, but could it be transferred across classes?
To answer this question, we test the transferability of patterns.
Since classes are related (e.g., cat andtiger), transferable patterns activated in one class could also be activated in other classes (e.g., felidpatterns).
Therefore, we ﬁrst ﬁnd important patterns for each base class by the classiﬁcation weights,then record activation of these patterns on other classes, and measure the transferability of patternsby the mean value of such other-class-activation.
The results are plotted in Fig.
2 (right).
As can beseen, the transferability consistently decreases when the margin increases.
Combine this result with
Fig.
2 (mid), we hold that patterns tend to be less transferable when they ﬁt each base class better.
Discussion.
The ﬁtness also reﬂects the how much the given pattern is speciﬁc to a base class.
Imagine the extreme situation where each base-class only needs one pattern for representation, theﬁtness would reach its upper bound to make such pattern thoroughly speciﬁc to the correspondingclass.
Therefore, we interpret that the higher the margin is, the more speciﬁc (overﬁtting) the patternsare to each base class, which makes patterns more discriminative and less transferable.
Meanwhile,the lower the margin is, the more the patterns could be shared between classes (underﬁtting), makingpatterns more transferable but less discriminative.
The CO dilemma lies in that patterns can hardly beboth class-speciﬁc and shared among classes by simply applying the classiﬁcation margin.
2.3.2 Inherent Class
Relations Lead to the Change in Pattern’s Base -Class Fitness
Pattern’s ﬁtness negatively inﬂuences class relations.
In Fig.
2 (right), we also plot the classrelations w.r.t.
the margins.
The class relations are measured by the average of cosine similaritiesbetween every two classes’ prototypes.
As can be seen, the relation drops as the margin grows, inconsistent with the trend of the patterns’ transferability.
This is rationale because the if two prototypesshare some patterns, the activation of the corresponding channels will be similar, making the cosinesimilarity larger.
As the transferability of patterns is negatively related to pattern’s base-class ﬁtness,we hold that the class relations are also negatively related to the base-class ﬁtness.
Inherent class relations inﬂuence pattern’s ﬁtness.
The margin applied to the classiﬁcationdirectly modiﬁes the decision boundary between every two classes, and the decision boundary isrelated to the relationship between every two classes.
Therefore, we study how the class relationinﬂuences the pattern’s ﬁtness to base classes.
Speciﬁcally, given 60 base classes, for the modeltrained without margins, we ﬁrst calculate the cosine similarity between every two different classes,which gives 60 × (60 - 1) / 2 = 1,750 relations denoted as R 0 , which represents the inherent relationsbetween all classes.
Similarly, we calculate 1,750 relations for the model trained with positive andnegative margins respectively, denoted as R pos and R neg .
Then we calculate D pos
=
R pos − R 0 and
D neg =
R neg − R 0 .
Finally, we sort R 0 from small to large, and use the sorted index to arrange D negand D pos .
As plotted in Fig.
3, the blue dots denote D neg (left) or D pos
(right), and the blue curve aredots ﬁtted by multinomial functions.
We can see that the change in class relations is positively relatedto the inherent class relations for the negative margins, while is negatively related to the inherentclass relations for the positive margins, especially for index larger than 1,000.
Since class relations are negatively related to pattern’s ﬁtness to base classes, the more the classrelation increases, the less the patterns could ﬁt the corresponding base class.
Therefore, the resultscan be understood that the more inherently similar two classes are, by applying a negative margin,the less the patterns could ﬁt the given base class, i.e., the more the patterns are shared by givenclasses, making these classes’ representations more similar; by applying a positive margin, the morethe patterns could ﬁt the given base classes, making these classes’ representations more dissimilar.
Conclusion and Discussion.
Therefore, as shown in Fig. 4, we interpret the pattern learning processas follows.
Given a negative margin, the decision boundary between two classes are confused, makingmore samples fall into the overlapping region between two classes.
This makes it possible to learnpatterns shared by these two classes and hinder the pattern from ﬁtting the given base class, andmakes patterns more transferable but less discriminative.
The more similar inherently two classesare (i.e., larger overlapping region in Fig. 4), the more shared patterns can be learned (e.g., morepatterns could be shared between cats and tigers than cats and air-planes), therefore making theseclasses’ representations more similar.
On the contrary, given a positive margin, the decision boundarybetween two classes should be well separated, pushing the model to learn patterns ﬁtting (i.e., speciﬁcto) each class, which are more discriminative but less transferable.
The more similar inherently twoclasses are, the harder the learning is and the larger the training loss will be, therefore making thepatterns ﬁt each class more and making these classes’ representations more dissimilar.
Mitigating the Dilemma of Few-Shot Class-Incremental Learning
In this section, we ﬁrst analyze the cause of the dilemma in margin-based classiﬁcation based onthe interpretation from the pattern learning view, then we propose our method to mitigate the COdilemma based on the analysis.
The framework is shown in Fig.
3.1 Analysis on the Cause of Class-Level Overﬁtting
Based on the above analysis, we can ﬁnd the learning of negative-margin-based patterns looselyconstrains the given pattern to be shared by both classes.
However, such constraint is easily satisﬁedby ineffective patterns such as edges or corners, which could lead to the low discriminability ofnegative-margin-based patterns.
Similarly, the learning of positive-margin-based patterns looselyconstrains the given pattern to be speciﬁc to the given class.
However, such constraint could besatisﬁed by easily ﬁnding patterns sharing no information with other classes, such as ﬁnding somecomplex texture only speciﬁc to the given class, which could lead to the low transferability of positivemargin-based patterns.
Such easy-constraint problems push patterns to be only class-speciﬁc orshared among classes, making pattern effective in one scenario ineffective in other ones.
Therefore,the key to mitigate the CO dilemma is to provide extra constraint to the pattern learning process.
3.2 Mitigating Class-Level Overﬁtting by Providing Extra Constraint
Negative-Margin-based (NM) patterns are more transferable and class-shared, while Positive-Marginbased (PM) patterns are discriminative and class-speciﬁc.
These characteristics are similar tothe behavior of features from the shallow and deep layers of deep networks [25, 30].
Generally,shallow-layer features encode low-level patterns that are easily shared by most objects therefore more transferable, such as edge or corner, while the deep-layer features encode high-level patterns thatare semantically related to only few object classes therefore more discriminative. Such similarityinspires us to view PM patterns as high-level patterns while view NM patterns as (relatively) low-levelpatterns, and build PM patterns from the NM patterns, just like building high-level features fromlow-level features in deep networks. As the learning of NM patterns is inﬂuenced by PM patterns,this design could beneﬁt the learning of NM patterns by providing extra constraint from the learningof PM patterns, and vice versa, which could therefore handle the easy-constraint problem.
Speciﬁcally, given an feature extractor f ( · ) , we add one more layer to construct the PM feature as
[Equation]
(3) where F ( · ) is the PM feature extractor, g ( · ) is typically composed of a fully-connected layer, a 211 batch-normalization layer [12] and an activation layer.
During training, two classiﬁcation loss isapplied to both F ( · ) and f ( · ) with positive and negative margins respectively, represented as L
[Equation]
x i , y i ) =
L neg ( x i
[Equation]
y i )
+
[Equation]
( x
[Equation]
y
[Equation]
[Equation]
where w Pj ∈ R 1 ×
d P denotes the classiﬁer weights for F ( · ) corresponding to class j , m neg denotesthe negative margin for f ( · ) , m pos is the positive margin for F ( · ) , and λ is typically set to 1.0.
Given this modiﬁcation, the NM patterns f ( · ) are utilized to construct the discriminative PM patterns
F ( · ) .
Since g ( · ) is relatively too simple to capture complex patterns, f ( · ) will be pushed to bediscriminative (validated in Fig.
7), instead of casually learning ineffective patterns shared betweensimilar classes such as edges or corners.
Similarly, as F ( · ) are built from the transferable f ( · ) by thesimple g ( · ) , the transferability could be maintained by pushing PM patterns to ﬁt the correspondingbase class by transferable information, which improves the transferability of F ( · ) (validated in Fig.
During testing, given the improved f ( · ) and F ( · ) , the ﬁnal feature would be their concatenation.
3.3 Further Mitigation of Class-Level Overﬁtting by Integrating Class Relations
Based on the architecture design, we propose to further mitigate the CO problem through boostingthe discriminability and tranferability of NM and PM patterns by integrating the class relations.
For easy understanding, we ﬁrst introduce the modiﬁcation to the classiﬁcation for f ( · ) , and laterintroduce that for F ( · ) .
To modify the margin-based classiﬁcation from a single margin to marginsrelated to class relations, we ﬁrst move the margin from the ground-truth logit to all other logits as
[Equation]
(6) where A is the adjacency matrix between all classes, m ( · ) maps the adjacency value between twoclasses to a margin.
Given this modiﬁcation, if the margin is set to the original ﬁxed value as Eq.
2,the decision boundary between class y i and other classes remains the same, therefore the new loss in
Eq.
6 could be an approximation of the original loss in Eq. 2.
For measuring class relations, we choose to utilize the adjacency matrix of all classes.
As both thefeature and the classiﬁer’s weights are L 2 normalized, the adjacency can be measured by the cosine
[Table]
[Table]
similarity between every two class prototypes as A ij = cos ( P i , P j ) , where P i is the prototype forclass i , which is typically set to w i of the classiﬁer.
If two classes are identical, the cosine similarity would reach its upper bound to 1.0.
Meanwhile, theaverage of class relations reﬂects a global margin that is effective for most classes.
Therefore, wedesign to interpolate the margin from a global effective value to a pre-deﬁned upper bound value as
[Equation]
where m upper is a hyper-parameter controlling the margin for the upper-bound class relation (i.e.,two identical classes), m ave is another hyper-parameter controlling the margin for the average classrelations, set to the same value as that in Eq. 2, and A ave is the average class relations calculated as
[Equation]
For F ( · ) , we adopt the same modiﬁcation as f ( · ) , which replaces f ( · ) with F ( · ) in Eq. 6, and replaces
A with A P in Eq.
6 where A Pij = cos ( w Pi , w Pj ) .
Therefore, the ﬁnal training objective is
L ( x i , y i )
= − log
[Equation]
where m p ( · ) and m n ( · ) are two interpolate functions as Eq.
7 with positive and negative hyperparameters ( m Pupper , m Pave ) and ( m upper , m ave ) respectively.
In experiments (Fig. 9), we ﬁnd it beneﬁcial to have m upper < m ave and m Pupper > m Pave , whichmeans for classes with higher similarities, the relation mapping module enables the model to learnmore shared (transferable) patterns by applying smaller negative margins, and learn more classspeciﬁc (discriminative) patterns by applying larger positive margins.
Moreover, due to the connectionbetween NM and PM patterns, the improved transferability in NM patterns would be transmitted to
PM patterns and vice versa, as validated in Fig.
9, which further mitigates the CO dilemma.
After the base-session training, the feature extractor F ( · ) and f ( · ) will be applied to the training dataof each session to obtain the extended classiﬁer weight W = { w 01 , w 02 , ..., w 0 | Y 0 | }∪ ... ∪{ w k 1 , ..., w k | Y k | }by averaging extracted features, and the ﬁnal performance of each session will be obtained based onthe classiﬁcation of all the encountered classes’ test samples.
Experiments
In this section, we ﬁrst introduce the experiment settings, then compare the proposed method withthe state-of-the-art methods, and ﬁnally report the ablation study for the effectiveness of each design.
4.1 Datasets
Datasets include CIFAR100 [14], Caltech-UCSD Birds-200-2011 (CUB200) [23] and miniImageNet [22] as listed in Tab. 1 following the split in [21].
For details, please refer to the appendix.
[Table]
4.2 Implementation Details
The implementation is based on CEC’s code [26], and our code will be released.
For CIFAR100, weset d P =256, set m ave =
-0.2, set m upper =
-0.5, and we have m Pave =0.1 and m Pupper =0.2.
For CUB200,we scale the learning rate of the backbone network to 10% of the global learning rate since thepre-training of the backbone is adopted [28, 26], and set d P to 8192.
Then we have m ave
=-0.2and m upper =
-0.25 and m Pave =0.3 and m Pupper =0.6.
For mini ImageNet, we set d P to 4096, and havem ave =
-0.2 and m upper =
-0.5 and m Pave =0.1 and m Pupper =0.2.
All models are optimized by the SGDoptimizer with momentum set to 0.9.
Please refer to the appendix for details.
4.3 Comparison with the State-of-the-Art
Comparisons with the state-of-the-art works are listed in Tab. 2 and Fig.
6, where we can achievestate-of-the-art performance on all three datasets.
Speciﬁcally, we can ﬁrst see that our method, as aprototype-based method (e.g., CEC [26]), could signiﬁcantly outperform ﬁnetune-based methods(e.g.
, iCaRL [17]).
This is because the few-shot training data could not provide sufﬁcient informationfor novel-class learning, therefore directly freezing parameters on novel classes could reduce thecatastrophic-forgetting problem brought by the ﬁnetuning.
Our method also outperforms otherprototype-based methods, this is because the core of the prototype-based methods lies in the metriclearning [19].
As empirically proved by current works [20, 24, 8, 15], applying the margin-basedclassiﬁcation could effectively improve the embedding space learned by metric-based methods.
Sinceour method handles the difﬁculty of applying margin-based classiﬁcation (i.e., class-level overﬁtting)to the FSCIL task, our method could beat these prototype-based ones.
For detailed numbers of
CIFAR100 and mini ImageNet, please refer to the appendix.
4.4 Ablation Study of Modules
The ablation study of each module is reported in Tab. 3, where g ( · ) denotes adding another layer,
Margin denotes the applicant of margin-based classiﬁcation, and Relation refers to the relationmapping of the margin.
We study from three aspects: Base -class, Novel -class, and Overall accuracyof the last incremental session.
From Tab. 3, we can see that
• Simply applying another layer cannot consistently improve the performance .
• The designed architecture could mitigate class-level overﬁtting .
Compared with experiments inFig.
2 where no Overall performance improvements can be obtained by simply adding margins, the performance here is clearly improved by adding margins on the designed architecture.
Moreover, the
overall improvements originate from not only the improved Base performance, but also the boosted
Novel performance, demonstrating the mitigation of class-level overﬁtting (CO) problem.
• Relation mapping could further improve performance by mitigating class-level overﬁtting .
4.5 Veriﬁcation of Class-Level Overﬁtting Mitigation
Mitigating the class-level overﬁtting by the architecture design.
We ﬁrst compare our methodwith the baseline method which applies the margin directly to the backbone feature in Fig.
7 and
8, so as to validate the mitigation of CO.
In Fig.
7, we apply the negative margin to the backbone(NM) feature while applying no margin to the PM feature, and see the accuracy of the NM featureagainst the baseline.
As can be seen, while keeping the improvements on novel classes (mid), ourmethod (orange) maintains the base-class accuracy (right), in contrast to a sharp decrease of thebaseline (blue) when the margin decreases, which makes our NM feature signiﬁcantly outperformsthe baseline in terms of the overall accuracy (left).
Similarly, in Fig.
8, our PM feature also maintainsa higher performance on novel classes when the margin increases.
These experiments validate themitigation of the CO dilemma: by implicitly guiding the learning of the PM and NM features by eachother, the transferability or discriminability drop on them can be mitigated respectively.
Further mitigating class-level overﬁtting by relation mapping.
To validate the relation mappingmodule, in Fig.
9 we ﬁxed the average margin (i.e., m ave =-0.2 in Fig.
9.1st and m Pave =0.3 in
Fig.
9.2nd) and experiment with different upper margins (i.e., m upper in Fig.
9.1st and m Pupper in
Fig.
9.2nd).
Similar to Fig.
7, when conducting experiments on one branch, the hyper-parameters ofthe other branch are ﬁxed.
We can see the relation mapping module can indeed help the learning, asthe upper margins are signiﬁcantly different from the average margin.
We also plot the NM and PMperformance on base classes with negative margins in Fig.
9(3rd), and those on novel classes withpositive margins in Fig.
9(4th).
We can see the NM and PM performance are improved simultaneously,which validates the improvements brought by the extra guidance of each features.
Therefore, therelation mapping module can also help to mitigate the class-level overﬁtting problem by building
PM features with more transferable patterns, and vice versa.
Moreover, it is interesting to ﬁnd theperformance is improved when m upper < m ave and m Pupper > m Pave , which means for classes withhigher similarities, the relation mapping module enables the model to learn more shared patternsby applying negative margins with larger absolute value, and learn more class-speciﬁc patterns byapplying larger positive margins.
Also, this is consistent with the results in Fig.
Conclusion
In this paper, we focus on the dilemma in the margin-based classiﬁcation for FSCIL.
We ﬁrst interpretthe dilemma as a class-level overﬁtting problem from the aspect of pattern learning, then ﬁnd thecause of this problem lies in the easily-satisﬁed constraint of learning shared or class-speciﬁc patterns.
Based on the analysis, we design a method to mitigate the dilemma by constructing PM patterns from
NM patterns, and mapping class relations into class-speciﬁc patterns.
Extensive experiments on threepublic datasets validate the effectiveness and outstanding performance of the proposed method.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[Yes](d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]2.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A](b)
Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes]
(c) Did you report error bars (e g., with respect to the random seed after running experiments multiple times)?
[No] Experiments are conducted based on the code released by CEC (CVPR’21), whichspeciﬁed the random seed for re-implementation and comparison.
Therefore, we follow the same random seed to conduct experiments and compare with other methods without
the error bar.(d) Did you include the total amount of compute and the type of resources used (e g., typeof GPUs, internal cluster, or cloud provider)?
[Yes]
If you are using existing assets (e g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[Yes]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[Yes]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[Yes]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]