Original: S UBSPACE R EGULARIZERS FOR F EW -S HOT C LASS I NCREMENTAL L EARNING
Revised: S UBSPACE R EGULARIZERS FOR F EW -S HOT C LASS I NCREMENTAL L EARNING

Original: Jacob Andreas
Revised: Jacob Andreas

Original: A BSTRACT
Revised: A BSTRACT

Original: Few-shot class incremental learning—the problem of updating a trained classiﬁer to discriminate among an expanded set of classes with limited labeled data—is a key challenge for machine learning systems deployed in non-stationary environments.
Revised: Few-shot class incremental learning—the problem of updating a trained classiﬁer to discriminate among an expanded set of classes with limited labeled data—is a key challenge for machine learning systems deployed in non-stationary environments.

Original: Existing approaches to the problem rely on complex model architectures and training procedures that are difﬁcult to tune and re-use.
Revised: Existing approaches to the problem rely on complex model architectures and training procedures that are difﬁcult to tune and re-use.

Original: In this paper, we present an extremely simple approach that enables the use of ordinary logistic regression classiﬁers for few-shot incremental learning.
Revised: In this paper, we present an extremely simple approach that enables the use of ordinary logistic regression classiﬁers for few-shot incremental learning.

Original: The key to this approach is a new family of subspace regularization schemes that encourage weight vectors for new classes to lie close to the subspace spanned by the weights of existing classes.
Revised: The key to this approach is a new family of subspace regularization schemes that encourage weight vectors for new classes to lie close to the subspace spanned by the weights of existing classes.

Original: When combined with pretrained convolutional feature extractors, logistic regression models trained with subspace regularization outperform specialized, state-of-the-art approaches to few-shot incremental image classiﬁcation by up to 22% on the mini ImageNet dataset.
Revised: When combined with pretrained convolutional feature extractors, logistic regression models trained with subspace regularization outperform specialized, state-of-the-art approaches to few-shot incremental image classiﬁcation by up to 23% on the mini ImageNet dataset.

Original: Because of its simplicity, subspace regularization can be straightforwardly extended to incorporate additional background information about the new classes (including class names and descriptions speciﬁed in natural language); these further improve accuracy by up to 2%.
Revised: Because of its simplicity, subspace regularization can be straightforwardly conﬁgured to incorporate additional background information about the new classes (including class names and descriptions speciﬁed in natural language); this offers additional control over the trade-off between existing and new classes.

Original: Our results show that simple geometric regularization of class representations offers an effective tool for continual learning.
Revised: Our results show that simple geometric regularization of class representations offers an effective tool for continual learning.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Standard approaches to classiﬁcation in machine learning assume a ﬁxed training dataset and a ﬁxed set of class labels.
Revised: Standard approaches to classiﬁcation in machine learning assume a ﬁxed training dataset and a ﬁxed set of class labels.

Original: But for many real-world classiﬁcation problems, these assumptions are unrealistic.
Revised: But for many real-world classiﬁcation problems, these assumptions are unrealistic.

Original: Classiﬁers must sometimes be updated on-the-ﬂy to recognize new concepts (e.g. new skills in personal assistants or new road signs in self-driving vehicles), while training data is sometimes unavailable for reuse (e.g. due to privacy regulations, Lesort et al. 2019;
Revised: Classiﬁers must sometimes be updated on-the-ﬂy to recognize new concepts (e.g. new skills in personal assistants or new road signs in self-driving vehicles), while training data is sometimes unavailable for reuse (e.g. due to privacy regulations, Lesort et al. 2019;

Original: McClure et al. 2018; or storage and retraining costs, Bender et al. 2021).
Revised: McClure et al. 2018; or storage and retraining costs, Bender et al. 2021).

Original: Development of models that support few-shot class-incremental learning (FSCIL), in which classiﬁers’ label sets can be easily extended with small numbers of new examples and no retraining, is a key challenge for machine learning systems deployed in the real world (Masana et al., 2020).
Revised: Development of models that support few-shot class-incremental learning (FSCIL), in which classiﬁers’ label sets can be easily extended with small numbers of new examples and no retraining, is a key challenge for machine learning systems deployed in the real world (Masana et al., 2020).

Original: As a concrete example, consider the classiﬁcation problem depicted in Fig.
Revised: As a concrete example, consider the classiﬁcation problem depicted in Fig.

Original: A model, initially trained on a large set of examples from several base classes ( snorkel , arctic fox , meerkat ; Fig.
Revised: A model, initially trained on a large set of examples from several base classes ( snorkel , arctic fox , meerkat ; Fig.

Original: 1a), must subsequently be updated to additionally recognize two novel classes ( white wolf and poncho ;
Revised: 1a), must subsequently be updated to additionally recognize two novel classes ( white wolf and poncho ;

Original: Fig. 1b), and ultimately distinguish among all ﬁve classes (Fig. 1c).
Revised: Fig. 1b), and ultimately distinguish among all ﬁve classes (Fig. 1c).

Original: Training a model to recognize the base classes is straightforward: for example, we can jointly optimize the parameters of a feature extractor (perhaps a convolutional network parameterized by θ ) and a linear classiﬁcation layer ( η ) to maximize the regularized likelihood of (image, label) pairs from the dataset in Fig.
Revised: Training a model to recognize the base classes is straightforward: for example, we can jointly optimize the parameters of a feature extractor (perhaps a convolutional network parameterized by θ ) and a linear classiﬁcation layer ( η ) to maximize the regularized likelihood of (image, label) pairs from the dataset in Fig.

Original: 1a:
Revised: 1a:

Original: [Equation]
Revised: [Equation]

Original: But how can this model be updated to additionally recognize the classes in Fig.
Revised: But how can this model be updated to additionally recognize the classes in Fig.

Original: 1b, with only a few examples of each new class and no access to the original training data?
Revised: 1b, with only a few examples of each new class and no access to the original training data?

Original: Na ¨ ıvely continuing to optimize Eq.
Revised: Na ¨ ıvely continuing to optimize Eq.

Original: 1 on (x, y) pairs drawn from the new dataset will cause several problems.
Revised: 1 on (x, y) pairs drawn from the new dataset will cause several problems.

Original: In the absence of any positive examples of those classes, performance on base classes will suffer due to catastrophic forgetting (Goodfellow et al., 2013), while performance on novel classes will likely be poor as a result of overﬁtting (Anderson & Burnham, 2004).
Revised: In the absence of any positive examples of those classes, performance on base classes will suffer due to catastrophic forgetting (Goodfellow et al., 2013), while performance on novel classes will likely be poor as a result of overﬁtting (Anderson & Burnham, 2004).

Original: As a consequence, most past work on FSCIL has focused on alternative approaches that use nonstandard prediction architectures (e g., Tao et al., 2020b) or optimize non-likelihood objectives (e g.,
Revised: As a consequence most past work on FSCIL has focused on alternative approaches that use nonstandard prediction architectures (e g., Tao et al., 2020b) or optimize non-likelihood objectives (e g.,

Original: Yoon et al., 2020; Ren et al., 2019).
Revised: Yoon et al., 2020; Ren et al., 2019).

Original: This divergence between approaches to standard and incremental classiﬁcation has its own costs—state-of-the-art approaches to FSCIL are complicated, requiring nested optimizers, complex data structures, and numerous hyperparameters.
Revised: This divergence between approaches to standard and incremental classiﬁcation has its own costs—state-of-the-art approaches to FSCIL are complicated, requiring nested optimizers, complex data structures, and numerous hyperparameters.

Original: When improved representation learning and optimization techniques are developed for standard classiﬁcation problems, it is often unclear to how to apply these to the incremental setting.
Revised: When improved representation learning and optimization techniques are developed for standard classiﬁcation problems, it is often unclear to how to apply these to the incremental setting.

Original: In this paper, we turn the standard approach to classiﬁcation into a surprisingly effective tool for
Revised: In this paper, we turn the standard approach to classiﬁcation into a surprisingly effective tool for

Original: FSCIL.
Revised: FSCIL.

Original: Speciﬁcally, we show that both catastrophic forgetting and overﬁtting can be reduced by introducing an additional subspace regularizer (related to one studied by Agarwal et al.and Kirkpatrick et al. 2017) that encourages novel η to lie close to the subspace spanned by the base classes.
Revised: Speciﬁcally, we show that both catastrophic forgetting and overﬁtting can be reduced by introducing an additional subspace regularizer (related to one studied by Agarwal et al.and Kirkpatrick et al. 2017) that encourages novel η to lie close to the subspace spanned by the base classes.

Original: On its own, the proposed subspace regularizer produces ordinary linear classiﬁers that achieve state-of-the-art results on FSCIL, improving over existing work in multiple tasks and datasets.
Revised: On its own, the proposed subspace regularizer produces ordinary linear classiﬁers that achieve state-of-the-art results on FSCIL, improving over existing work in multiple tasks and datasets.

Original: Because of its simplicity, this regularization approach can be easily extended to incorporate additional information about relationships between base and novel classes.
Revised: Because of its simplicity, this regularization approach can be easily conﬁgured to incorporate additional information about relationships between base and novel classes.

Original: Using language data as a source of background knowledge about classes, we describe a variation of our approach, which we term semantic subspace regularization , that pulls weight vectors toward particular convex combinations of base classes that capture their semantic similarity to existing classes.
Revised: Using language data as a source of background knowledge about classes, we describe a variation of our approach, which we term semantic subspace regularization , that pulls weight vectors toward particular convex combinations of base classes that capture their semantic similarity to existing classes.

Original: This further improves accuracy by up to2% over simple subspace regularization across multiple tasks.
Revised: Semantic subspace regularization results in a comparable performance to subspace regularization on average; it better preserves performance on existing classes at the expense of higher accuracy on newer classes.

Original: These results suggest that FSCIL and related problems may not require specialized machinery to solve, and that simple regularization approaches can solve the problems that result from limited access to training data for both base and novel classes.
Revised: These results suggest that FSCIL and related problems may not require specialized machinery to solve, and that simple regularization approaches can solve the problems that result from limited access to training data for both base and novel classes.

Original:  B ACKGROUND
Revised:  B ACKGROUND

Original: A long line of research has focused on the development of automated decision-making systems that support online expansion of the set of concepts they can recognize and generate.
Revised: A long line of research has focused on the development of automated decision-making systems that support online expansion of the set of concepts they can recognize and generate.

Original: An early example (closely related to our learning-from-deﬁnitions experiment in Section 5) appears in the classic SHRDLU language grounding environment (Winograd, 1972): given the deﬁnition a steeple is a small triangle on top of a tall rectangle , SHRDLU acquires the ability to answer questions containing the novel concept steeple .
Revised: An early example (closely related to our learning-from-deﬁnitions experiment in Section 5) appears in the classic SHRDLU language grounding environment (Winograd, 1972): given the deﬁnition a steeple is a small triangle on top of a tall rectangle , SHRDLU acquires the ability to answer questions containing the novel concept steeple .

Original: Recent work in machine learning describes several versions of this problem in featuring more complex perception or control:
Revised: Recent work in machine learning describes several versions of this problem in featuring more complex perception or control:

Original: Few-shot and incremental learning Few-shot classiﬁcation problems test learners’ ability to distinguish among a ﬁxed set of classes using only a handful of labeled examples per class (Scheirer et al., 2012).
Revised: Few-shot and incremental learning Few-shot classiﬁcation problems test learners’ ability to distinguish among a ﬁxed set of classes using only a handful of labeled examples per class (Scheirer et al., 2012).

Original: Most effective approaches to few-shot learning rely on additional data for pre-training (Tian et al., 2020) or meta-learning (Vinyals et al., 2016;
Revised: Most effective approaches to few-shot learning rely on additional data for pre-training (Tian et al., 2020) or meta-learning (Vinyals et al., 2016;

Original: Finn et al., 2017; Snell et al., 2017; Yoon et al., 2019).
Revised: Finn et al., 2017; Snell et al., 2017; Yoon et al., 2019).

Original: One peculiarity of this evaluation paradigm is that, even when pre-trained, models are evaluated only on new (few-shot) classes, and free to update their parameters in ways that cause them to perform poorly on pre-training tasks.
Revised: One peculiarity of this evaluation paradigm is that, even when pre-trained, models are evaluated only on new (few-shot) classes, and free to update their parameters in ways that cause them to perform poorly on pre-training tasks.

Original: As noted by past work (Kuzborskij et al., 2013), a more realistic evaluation of models’ ability to rapidly acquire new concepts should consider their ability to discriminate among both new concepts and old ones,
Revised: As noted by past work (Kuzborskij et al., 2013), a more realistic evaluation of models’ ability to rapidly acquire new concepts should consider their ability to discriminate among both new concepts and old ones,

Original: a problem usually referred to as few-shot class-incremental learning (FSCIL) 

Revised: a problem usually referred to as few-shot class-incremental learning (FSCIL)

Original: (Tao et al., 2020b).
Revised: (Tao et al., 2020b).

Original: FSCIL requires learners to incrementally acquire novel classes with few labeled examples while retaining high accuracy on previously learned classes.
Revised: FSCIL requires learners to incrementally acquire novel classes with few labeled examples while retaining high accuracy on previously learned classes.

Original: It combines the most challenging aspects of class-incremental learning (Rebufﬁ et al., 2017) task-incremental learning (Delange et al., 2021), and rehearsal-based learning (Rolnick et al., 2019; Chaudhry et al., 2019), three related problems with much stronger assumptions about the kind of information available to learners.
Revised: It combines the most challenging aspects of class-incremental learning (Rebufﬁ et al., 2017) task-incremental learning (Delange et al., 2021), and rehearsal-based learning (Rolnick et al., 2019; Chaudhry et al., 2019), three related problems with much stronger assumptions about the kind of information available to learners.

Original: Existing approaches to this problem either prioritize novel class adaptation (Ren et al., 2019; Yoon et al., 2020;
Revised: Existing approaches to this problem either prioritize novel class adaptation (Ren et al., 2019; Yoon et al., 2020;

Original: Chen & Lee, 2021; Cheraghian et al., 2021) or reducing forgetting in old classes (Tao et al., 2020b).
Revised: Chen & Lee, 2021; Cheraghian et al., 2021) or reducing forgetting in old classes (Tao et al., 2020b).

Original: Learning class representations Even prior to the widespread use of deep representation learning approaches, the view of classiﬁcation as problem of learning class representations motivated a number of approaches to multi-class and multi-task learning (Argyriou et al., 2007a; Agarwal et al., 2010).
Revised: Learning class representations Even prior to the widespread use of deep representation learning approaches, the view of classiﬁcation as problem of learning class representations motivated a number of approaches to multi-class and multi-task learning (Argyriou et al., 2007a; Agarwal et al., 2010).

Original: In few-shot and incremental learning settings, many recent approaches have also focused on the space of class representations (Tao et al., 2020a).
Revised: In few-shot and incremental learning settings, many recent approaches have also focused on the space of class representations (Tao et al., 2020a).

Original: Qi et al.
Revised: Qi et al.

Original: (2018) initialize novel class representations using the average features from few-shot samples.
Revised: (2018) initialize novel class representations using the average features from few-shot samples.

Original: Others (Gidaris & Komodakis, 2018;
Revised: Others (Gidaris & Komodakis, 2018;

Original: Yoon et al., 2020; Zhang et al., 2021) train a class representation predictor via meta-learning, and
Revised: Yoon et al., 2020; Zhang et al., 2021) train a class representation predictor via meta-learning, and

Original: Tao et al.
Revised: Tao et al.

Original: (2020b) impose topological constraints on the manifold of class representations as new representations are added.
Revised: (2020b) impose topological constraints on the manifold of class representations as new representations are added.

Original: Alternatively, Chen & Lee (2021) models the visual feature space as a
Revised: Alternatively, Chen & Lee (2021) models the visual feature space as a

Original: Gaussian mixture and use the cluster centers in a similarity-based classiﬁcation scheme.
Revised: Gaussian mixture and use the cluster centers in a similarity-based classiﬁcation scheme.

Original: Lastly, two concurrent works condition both old and new class representations at each session according to an auxiliary scheme; graph attention network in Zhang et al.
Revised: Lastly, two concurrent works condition both old and new class representations at each session according to an auxiliary scheme; graph attention network in Zhang et al.

Original: and relation projection in Zhu et al.
Revised: and relation projection in Zhu et al.

Original: Our approach is related to Ren et al.
Revised: Our approach is related to Ren et al.

Original: (2019), who uses a nested optimization framework to learn auxiliary parameters for every base and novel class to inﬂuence the novel weights via regularization; we show that these regularization targets can be derived geometrically without the need for an inner optimization step.
Revised: (2019), who uses a nested optimization framework to learn auxiliary parameters for every base and novel class to inﬂuence the novel weights via regularization; we show that these regularization targets can be derived geometrically without the need for an inner optimization step.

Original: Also related is the work of Barzilai & Crammer (2015), which synthesizes the novel weights as linear combinations of base weights; we adopt a regularization approach that allows learning of class representations that are not strict linear combinations of base classes.
Revised: Also related is the work of Barzilai & Crammer (2015), which synthesizes the novel weights as linear combinations of base weights; we adopt a regularization approach that allows learning of class representations that are not strict linear combinations of base classes.

Original: Moreover,
Revised: Moreover,

Original: Kuzborskij et al.
Revised: Kuzborskij et al.

Original: (2013) study a class incremental learning setup where they increment the number of classes by one.
Revised: (2013) study a class incremental learning setup where they increment the number of classes by one.

Original: Similar to ours, the parameters for the novel class is regularized towards a weighted combination of old class parameters while using as many examples from old classes as there are from novel classes.
Revised: Similar to ours, the parameters for the novel class is regularized towards a weighted combination of old class parameters while using as many examples from old classes as there are from novel classes.

Original: In comparison, our approach does not require any examples from old classes.
Revised: In comparison, our approach does not require any examples from old classes.

Original: Learning with side information from language The use of background information from other modalities (especially language) to bootstrap learning of new classes is widely studied (Frome et al.,
Revised: Learning with side information from language The use of background information from other modalities (especially language) to bootstrap learning of new classes is widely studied (Frome et al.,

Original: ; Radford et al., 2021;
Revised: ; Radford et al., 2021;

Original: Reed et al., 2016; Yan et al., 2021)—particularly in the zero-shot learning and generalized zero-shot learning where side information is the only source of information about the novel class (Chang et al., 2008; Larochelle et al., 2008; Akata et al., 2013; Pourpanah et al., 2020).
Revised: Reed et al., 2016; Yan et al., 2021)—particularly in the zero-shot learning and generalized zero-shot learning where side information is the only source of information about the novel class (Chang et al., 2008; Larochelle et al., 2008; Akata et al., 2013; Pourpanah et al., 2020).

Original: Specialized approaches exist for integrating side information in few-shot learning settings (Schwartz et al., 2019; Cheraghian et al., 2021).
Revised: Specialized approaches exist for integrating side information in few-shot learning settings (Schwartz et al., 2019; Cheraghian et al., 2021).

Original:  P ROBLEM F ORMULATION
Revised:  P ROBLEM F ORMULATION

Original: We follow the notation in Tao et al.
Revised: We follow the notation in Tao et al.

Original: (2020b) for FSCIL: assume a stream of T learning sessions , each associated with a labeled dataset D (0) , D (1) , . . .
Revised: (2020b) for FSCIL: assume a stream of T learning sessions , each associated with a labeled dataset D (0) , D (1) , . . .

Original: , D ( T ) .
Revised: , D ( T ) .

Original: Every D ( t ) consists of a support set
Revised: Every D ( t ) consists of a support set

Original: S ( t ) (used for training) and a query set Q ( t ) (used for evaluation).
Revised: S ( t ) (used for training) and a query set Q ( t ) (used for evaluation).

Original: We will refer to the classes represented in D (0) as base classes ; as in Fig.
Revised: We will refer to the classes represented in D (0) as base classes ; as in Fig.

Original: 1a, we will assume that it contains a large number of examples for every class.
Revised: 1a, we will assume that it contains a large number of examples for every class.

Original: D (1) (and subsequent datasets) introduce novel classes (Fig. 1b).
Revised: D (1) (and subsequent datasets) introduce novel classes (Fig. 1b).

Original: Let
Revised: Let

Original: C ( S ) =
Revised: C ( S ) =

Original: { y : ( x, y ) ∈ S } denote the set of classes expressed in a set of examples S ; we will write
Revised: { y : ( x, y ) ∈ S } denote the set of classes expressed in a set of examples S ; we will write

Original: C ( t ) =
Revised: C ( t ) =

Original: C ( S ( t ) ) and C ( ≤ t ) :
Revised: C ( S ( t ) ) and C ( ≤ t ) :

Original: = (cid:83) j ≤ t C ( S ( j ) ) for convenience.
Revised: = (cid:83) j ≤ t C ( S ( j ) ) for convenience.

Original: The learning problem we study is incremental in the sense that each support set contains only new classes ( C ( t ) ∩ C ( <t ) = ∅ ) 3 , while each query set evaluates models on both novel classes and previously seen ones ( C ( Q ( t ) )
Revised: The learning problem we study is incremental in the sense that each support set contains only new classes ( C ( t ) ∩ C ( <t ) = ∅ ) 3 , while each query set evaluates models on both novel classes and previously seen ones ( C ( Q ( t ) )

Original: =
Revised: =

Original: C ( ≤ t ) ).
Revised: C ( ≤ t ) ).

Original: It is few-shot in the sense that for t > 0 , | S ( t ) | is small (containing 1–5 examples for all datasets studied in this paper).
Revised: It is few-shot in the sense that for t > 0 , | S ( t ) | is small (containing 1–5 examples for all datasets studied in this paper).

Original: Given an incremental learning session t > 0 the goal is to ﬁne-tune existing classiﬁer with the limited training data from novel classes such that the classiﬁer performs well in classifying all classes learned thus far.
Revised: Given an incremental learning session t > 0 the goal is to ﬁne-tune existing classiﬁer with the limited training data from novel classes such that the classiﬁer performs well in classifying all classes learned thus far.

Original: FSCIL with a single session Prior to Tao et al.
Revised: FSCIL with a single session Prior to Tao et al.

Original: (2020b),
Revised: (2020b),

Original: a simpler version of the multi-session
Revised: a simpler version of the multi-session

Original: FSCIL was proposed by Qi et al.
Revised: FSCIL was proposed by Qi et al.

Original: (2018) where there is only single incremental learning session after the pre-training stage i.e. T = 1 .
Revised: (2018) where there is only single incremental learning session after the pre-training stage i.e. T = 1 .

Original: This version, which we call single-session FSCIL, has been extensively studied by previous work (Qi et al., 2018; Gidaris & Komodakis, 2018;
Revised: This version, which we call single-session FSCIL, has been extensively studied by previous work (Qi et al., 2018; Gidaris & Komodakis, 2018;

Original: Ren et al., 2019;
Revised: Ren et al., 2019;

Original: Yoon et al., 2020).
Revised: Yoon et al., 2020).

Original: This problem formulation is the same as above with T = 1 : a feature extractor is trained on the samples from D (0) , then D (1) , then evaluated on samples with classes in C (0) ∪ C (1) .
Revised: This problem formulation is the same as above with T = 1 : a feature extractor is trained on the samples from D (0) , then D (1) , then evaluated on samples with classes in C (0) ∪ C (1) .

Original:  A PPROACH
Revised:  A PPROACH

Original: Our approach to FSCIL consists of two steps.
Revised: Our approach to FSCIL consists of two steps.

Original: In the base session, we jointly train a feature extractor and classiﬁcation layer on base classes (Section 4.1).
Revised: In the base session, we jointly train a feature extractor and classiﬁcation layer on base classes (Section 4.1).

Original: In subsequent (incremental learning) sessions, we freeze the feature extractor and update only the classiﬁcation layer using regularizers that (1) stabilize representations of base classes, and (2) bring the representations of new classes close to existing ones (Sections 4.2-4.4).
Revised: In subsequent (incremental learning) sessions, we freeze the feature extractor and update only the classiﬁcation layer using regularizers that (1) stabilize representations of base classes, and (2) bring the representations of new classes close to existing ones (Sections 4.2-4.4).

Original: 4. F EATURE E XTRACTOR T RAINING
Revised: 4. F EATURE E XTRACTOR T RAINING

Original: As in Eq. 1, we begin by training an ordinary classiﬁer comprising a non-linear feature extractor f θ and a linear decision layer with parameters η .
Revised: As in Eq. 1, we begin by training an ordinary classiﬁer comprising a non-linear feature extractor f θ and a linear decision layer with parameters η .

Original: We choose η and θ to maximize:
Revised: We choose η and θ to maximize:

Original: [Equation]
Revised: [Equation]

Original: As discussed in Section 5, all experiments in this paper implement f θ as a convolutional neural network.
Revised: As discussed in Section 5, all experiments in this paper implement f θ as a convolutional neural network.

Original: In subsequent loss formulations we refer to (cid:107) η (cid:107) 2 + (cid:107) θ (cid:107) 

Revised: In subsequent loss formulations we refer to (cid:107) η (cid:107) 2 + (cid:107) θ (cid:107)

Original: as R prior ( η, θ ) .
Revised: as R prior ( η, θ ) .

Original: 4. F INE - TUNING
Revised: 4. F INE - TUNING

Original: Along with the estimated ˆ θ , feature extractor training yields parameters only for base classes η y ∈ C (0) .
Revised: Along with the estimated ˆ θ , feature extractor training yields parameters only for base classes η y ∈ C (0) .

Original: Given an incremental learning dataset D ( t ) , we introduce new weight vectors η c ∈ C ( t ) and optimize
Revised: Given an incremental learning dataset D ( t ) , we introduce new weight vectors η c ∈ C ( t ) and optimize

Original: [Equation]
Revised: [Equation]

Original: with respect to η alone.
Revised: with respect to η alone.

Original: Eq.
Revised: Eq.

Original: 3 features two new regularization terms, R ( t ) old and R ( t ) new .
Revised: 3 features two new regularization terms, R ( t ) old and R ( t ) new .

Original: R told which also appears in past work (Kuzborskij et al., 2013), limits the extent to which ﬁne-tuning can change parameters for classes that have already been learned:
Revised: R told which also appears in past work (Kuzborskij et al., 2013), limits the extent to which ﬁne-tuning can change parameters for classes that have already been learned:

Original: [Equation]
Revised: [Equation]

Original: where η t (cid:48)
Revised: where η t (cid:48)

Original: c denotes the value of the corresponding variable at the end of session t (cid:48) .
Revised: c denotes the value of the corresponding variable at the end of session t (cid:48) .

Original: (For example, η 0 c refers to the weights for the base class c prior to ﬁne tuning, i.e. after session t (cid:48)
Revised: (For example, η 0 c refers to the weights for the base class c prior to ﬁne tuning, i.e. after session t (cid:48)

Original: = 0 .)
Revised: = 0 .)

Original: As shown in
Revised: As shown in

Original: Section 5.2, using R old alone, and setting R new = 0 , is a surprisingly effective baseline; however, performance can be improved by appropriately regularizing new parameters as described below.
Revised: Section 5.2, using R old alone, and setting R new = 0 , is a surprisingly effective baseline; however, performance can be improved by appropriately regularizing new parameters as described below.

Original: Variant: Memory
Revised: Variant: Memory

Original: Following past work (Chen & Lee, 2021) which performs incremental learning while retaining a small “memory” of previous samples M , we explore an alternative baseline approach in which we append S ( t ) in Eq.
Revised: Following past work (Chen & Lee, 2021) which performs incremental learning while retaining a small “memory” of previous samples M , we explore an alternative baseline approach in which we append S ( t ) in Eq.

Original: 3 with M ( t ) .
Revised: 3 with M ( t ) .

Original: We deﬁne the memory at session t as
Revised: We deﬁne the memory at session t as

Original: M ( t ) =
Revised: M ( t ) =

Original: (cid:83) ( t (cid:48) <
Revised: (cid:83) ( t (cid:48) <

Original: t ) M ( t (cid:48) )
Revised: t ) M ( t (cid:48) )

Original: where M ( t (cid:48) )
Revised: where M ( t (cid:48) )

Original: ⊆ S ( t (cid:48) ) and | M ( t (cid:48) )
Revised: ⊆ S ( t (cid:48) ) and | M ( t (cid:48) )

Original: | = | C ( t (cid:48) ) | .
Revised: | = | C ( t (cid:48) ) | .

Original: We sample only 1 example per previous class and we reuse the same example in subsequent sessions.
Revised: We sample only 1 example per previous class and we reuse the same example in subsequent sessions.

Original: 4. M ETHOD 1: S UBSPACE R EGULARIZATION
Revised: 4. M ETHOD 1: S UBSPACE R EGULARIZATION

Original: Past work on other multitask learning problems has demonstrated the effectiveness of constraining parameters for related tasks to be similar (Jacob et al., 2008), lie on the same manifold (Agarwal et al., 2010) or even on the same linear subspace (Argyriou et al., 2007a).
Revised: Past work on other multitask learning problems has demonstrated the effectiveness of constraining parameters for related tasks to be similar (Jacob et al., 2008), lie on the same manifold (Agarwal et al., 2010) or even on the same linear subspace (Argyriou et al., 2007a).

Original: Moreover, Sch ¨ onfeld et al.
Revised: Moreover, Sch ¨ onfeld et al.

Original: (2019) showed that a shared latent feature space for all classes is useful for class-incremental classiﬁcation.
Revised: (2019) showed that a shared latent feature space for all classes is useful for class-incremental classiﬁcation.

Original: Features independently learned for novel classes from small numbers of examples are likely to capture spurious correlations (unrelated to the true causal structure of the prediction problem) as a result of dataset biases (Arjovsky et al., 2019).
Revised: Features independently learned for novel classes from small numbers of examples are likely to capture spurious correlations (unrelated to the true causal structure of the prediction problem) as a result of dataset biases (Arjovsky et al., 2019).

Original: In contrast, we expect most informative semantic features to be shared across multiple classes: indeed, cognitive research suggests that in humans’ early visual cortex, representations of different objects occupy a common feature space (Kriegeskorte et al., 2008).
Revised: In contrast, we expect most informative semantic features to be shared across multiple classes: indeed, cognitive research suggests that in humans’ early visual cortex, representations of different objects occupy a common feature space (Kriegeskorte et al., 2008).

Original: Therefore, regularizing toward the space spanned by base class weight vectors encourages new class representations to depend on semantic rather than spurious features and features for all tasks to lie in the same universal subspace.
Revised: Therefore, regularizing toward the space spanned by base class weight vectors encourages new class representations to depend on semantic rather than spurious features and features for all tasks to lie in the same universal subspace.

Original: We apply this intuition to FSCIL via a simple subspace regularization approach.
Revised: We apply this intuition to FSCIL via a simple subspace regularization approach.

Original: Given a parameter for an incremental class η c and base class parameters { η j ∈ C (0) } , we ﬁrst compute the subspace target m c for each class.
Revised: Given a parameter for an incremental class η c and base class parameters { η j ∈ C (0) } , we ﬁrst compute the subspace target m c for each class.

Original: We then compute the distance between η c from m c and deﬁne:
Revised: We then compute the distance between η c from m c and deﬁne:

Original: [Equation]
Revised: [Equation]

Original: where m c is the projection of η c onto the space spanned by { η j ∈ C (0) } :
Revised: where m c is the projection of η c onto the space spanned by { η j ∈ C (0) } :

Original: [Equation]
Revised: [Equation]

Original: and P C (0) contains the orthogonal basis vectors of the subspace spanned by the initial set of base weights η j ∈ C (0) .
Revised: and P C (0) contains the orthogonal basis vectors of the subspace spanned by the initial set of base weights η j ∈ C (0) .

Original: ( P C (0) can be found using a QR decomposition of the matrix of base class vectors, as described in the appendix.)
Revised: ( P C (0) can be found using a QR decomposition of the matrix of base class vectors, as described in the appendix.)

Original: Previous work that leverages subspace regularization for multitask learning assume that data from all tasks are available from the beginning (Argyriou et al., 2007b; Agarwal et al., 2010; Argyriou et al., 2007a).
Revised: Previous work that leverages subspace regularization for multitask learning assume that data from all tasks are available from the beginning (Argyriou et al., 2007b; Agarwal et al., 2010; Argyriou et al., 2007a).

Original: Our approach to subspace regularization removes these assumptions, enabling tasks (in this case, novel classes) to arrive incrementally and predictions to be made cumulatively over all classes seen thus far without any further information on which task that a query belongs to.
Revised: Our approach to subspace regularization removes these assumptions, enabling tasks (in this case, novel classes) to arrive incrementally and predictions to be made cumulatively over all classes seen thus far without any further information on which task that a query belongs to.

Original: Agarwal et al.
Revised: Agarwal et al.

Original: (2010) is similar to ours in encouraging all task parameters to lie on the same manifold; it is different in that they learn the manifold and the task parameters alternately.
Revised: (2010) is similar to ours in encouraging all task parameters to lie on the same manifold; it is different in that they learn the manifold and the task parameters alternately.

Original: Also related Simon et al.
Revised: Also related Simon et al.

Original: and Devos & Grossglauser (2019) model class representations over a set of subspaces (disjoint in the latter) for non-incremental few-shot learning.
Revised: and Devos & Grossglauser (2019) model class representations over a set of subspaces (disjoint in the latter) for non-incremental few-shot learning.

Original: 4. M ETHOD 2: S EMANTIC S UBSPACE R EGULARIZATION
Revised: 4. M ETHOD 2: S EMANTIC S UBSPACE R EGULARIZATION

Original: The constraint in Eq.
Revised: The constraint in Eq.

Original: 5 makes explicit use of geometric information about base classes, pulling novel weights toward the base subspace.
Revised: 5 makes explicit use of geometric information about base classes, pulling novel weights toward the base subspace.

Original: However, it provides no information about where within that subspace the weights for a new class should lie.
Revised: However, it provides no information about where within that subspace the weights for a new class should lie—potentially causing interference with base classes.

Original: In most classiﬁcation problems, classes have names consisting of natural language words or phrases; these names often contain a signiﬁcant amount of information relevant to the classiﬁcation problem of interest.
Revised: In most classiﬁcation problems, classes have names consisting of natural language words or phrases; these names often contain a signiﬁcant amount of information relevant to the classiﬁcation problem of interest.

Original: (Even without having ever seen a white wolf , a typical English speaker can guess that a white wolf is more likely to resemble an arctic fox than a snorkel .)
Revised: (Even without having ever seen a white wolf , a typical English speaker can guess that a white wolf is more likely to resemble an arctic fox than a snorkel .)

Original: These kinds of relations are often captured by embeddings of class labels (or more detailed class descriptions)
Revised: These kinds of relations are often captured by embeddings of class labels (or more detailed class descriptions)

Original: (Pennington et al., 2014).
Revised: (Pennington et al., 2014).

Original: When available, this kind of information about class semantics can be used to construct an improved subspace regularizer by encouraging new class representations to lie close to a convex combination of base classes weighted by their semantic similarity.
Revised: When available, this kind of information about class semantics can be used to construct an improved subspace regularizer by encouraging new class representations to lie close to a convex combination of base classes weighted by their semantic similarity.

Original: We replace the subspace projection P (cid:62) C (0)
Revised: We replace the subspace projection P (cid:62) C (0)

Original: η c in
Revised: η c in

Original: Eq.
Revised: Eq.

Original: 5 with a semantic target l c for each class.
Revised: 5 with a semantic target l c for each class.

Original: Letting e c denote a semantic embedding of the class c , we compute:
Revised: Letting e c denote a semantic embedding of the class c , we compute:

Original: [Equation]
Revised: [Equation]

Original: ) where
Revised: ) where

Original: [Equation]

Original: =

Original: [Equation]
Revised: [Equation]

Original: and τ is a hyper-parameter.
Revised: and τ is a hyper-parameter.

Original: Embeddings e c can be derived from multiple sources: in addition to the class names discussed above, a popular source of side information for zero-shot and few-shot learning problems is detailed textual descriptions of classes; we evaluate both label and description embeddings in Section 5.
Revised: Embeddings e c can be derived from multiple sources: in addition to the class names discussed above, a popular source of side information for zero-shot and few-shot learning problems is detailed textual descriptions of classes; we evaluate both label and description embeddings in Section 5.

Original: Sch ¨ onfeld et al.
Revised: Sch ¨ onfeld et al.

Original: (2019) also leverage label information on a shared subspace for few-shot incremental learning where they project both visual and semantic features onto a shared latent space for prediction in the single-session setting.
Revised: (2019) also leverage label information on a shared subspace for few-shot incremental learning where they project both visual and semantic features onto a shared latent space for prediction in the single-session setting.

Original: In comparison, we re-use the base visual space for joint projection for multiple incremental sessions.
Revised: In comparison, we re-use the base visual space for joint projection for multiple incremental sessions.

Original: Baseline: Linear Mapping While the approach described in Eq.
Revised: Baseline: Linear Mapping While the approach described in Eq.

Original: 7 combines semantic information and label subspace information, a number of previous studies in vision and language have also investigated the effectiveness of directly learning a mapping from the space of semantic embeddings to the space of class weights (Das & Lee, 2019; Socher et al., 2013; Pourpanah et al., 2020; RomeraParedes & Torr, 2015).
Revised: 7 combines semantic information and label subspace information, a number of previous studies in vision and language have also investigated the effectiveness of directly learning a mapping from the space of semantic embeddings to the space of class weights (Das & Lee, 2019; Socher et al., 2013; Pourpanah et al., 2020; RomeraParedes & Torr, 2015).

Original: Despite pervasiveness of the idea in other domains, this is the ﬁrst time we are aware of it being explored for FSCIL.
Revised: Despite pervasiveness of the idea in other domains, this is the ﬁrst time we are aware of it being explored for FSCIL.

Original: We extend our approach to incorporate this past work by learning a linear map L between the embedding space e j ∈ E and the weight space containing η 
Revised: We extend our approach to incorporate this past work by learning a linear map L between the embedding space e j ∈ E and the weight space containing η C

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: then set
Revised: then set

Original: [Equation]
Revised: [Equation]

Original: Concurrent work by (Cheraghian et al., 2021) also leverages side information for FSCIL where they learn a mapping from image space onto the label space to directly produce predictions in the label space.
Revised: Concurrent work by (Cheraghian et al., 2021) also leverages side information for FSCIL where they learn a mapping from image space onto the label space to directly produce predictions in the label space.

Original: We provide comparisons in Section 5.
Revised: We provide comparisons in Section 5.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: Given a classiﬁer trained on an initial set of base classes, our experiments aim to evaluate the effect of subspace regularization (1) on the learning of new classes, and (2) on the retention of base classes.
Revised: Given a classiﬁer trained on an initial set of base classes, our experiments aim to evaluate the effect of subspace regularization (1) on the learning of new classes, and (2) on the retention of base classes.

Original: To evaluate the generality of our method, we evaluate using two different experimental paradigms that have been used in past work: a multi-session experiment in which new classes are continuouslyadded and the classiﬁer must be repeatedly updated, and a single-session setup ( T = 1 ) in which new
Revised: To evaluate the generality of our method, we evaluate using two different experimental paradigms that have been used in past work: a multi-session experiment in which new classes are continuously

Original: [Figure]
Revised: [Figure]

Original:  classes arrive only once.
Revised: added and the classiﬁer must be repeatedly updated, and a single-session setup ( T = 1 ) in which new classes arrive only once.

Original: As our optimizer, we use stochastic gradient descent for feature extractor training and gradient descent in incremental sessions to train all models.
Revised: As our optimizer, we use stochastic gradient descent for feature extractor training and gradient descent in incremental sessions to train all models.

Original: Details about experiment setups and results are discussed below.
Revised: Details about experiment setups and results are discussed below.

Original: Additional details may be found in the appendix.
Revised: Additional details may be found in the appendix

Original: 5. M ULTI -S ESSION
Revised: 5. M ULTI -S ESSION

Original: memory distinction
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: We follow the same setup established in Tao et al
Revised: We follow the same setup established in Tao et al

Original: [Table] (2020b) as well as Section 3: we ﬁrst train a ResNet (He et al., 2016) network from scratch as the feature extractor on a large number of examples from base classes C (0) to obtain an initial classiﬁer η j ∈ C (0) .
Revised: (2020b) as well as Section 3: we ﬁrst train a ResNet (He et al., 2016) network from scratch as the feature extractor on a large number of examples from base classes C (0) to obtain an initial classiﬁer η j ∈ C (0) .

Original: We then observe a new batch of examples
Revised: We then observe a new batch of examples

Original: S ( t ) and produce a new classiﬁer deﬁned by η c ∈ C ( ≤ t ) .
Revised: S ( t ) and produce a new classiﬁer deﬁned by η c ∈ C ( ≤ t ) .

Original: Finally, we evaluate the classiﬁer according to top-1 accuracy in base and novel samples as well as their weighted average (Tao et al., 2020b; Chen & Lee, 2021).
Revised: Finally, we evaluate the classiﬁer according to top-1 accuracy in base and novel samples as well as their weighted average (Tao et al., 2020b; Chen & Lee, 2021).

Original: We use the mini ImageNet dataset (Vinyals et al., 2016; Russakovsky et al., 2015) for our multi-session evaluation.
Revised: We use the mini ImageNet dataset (Vinyals et al., 2016; Russakovsky et al., 2015) for our multi-session evaluation.

Original: mini ImageNet contains 100 classes with 600 samples per class.
Revised: mini ImageNet contains 100 classes with 600 samples per class.

Original: Feature extractor training In session t = 0 , we randomly select 60 classes as base classes ( | C (0) | = 60 ) and use the remaining 40 classes as novel classes.
Revised: Feature extractor training In session t = 0 , we randomly select 60 classes as base classes ( | C (0) | = 60 ) and use the remaining 40 classes as novel classes.

Original: Reported results are averaged across 10 random splits of the data (Fig. 2).
Revised: Reported results are averaged across 10 random splits of the data (Fig. 2).

Original: We use a ResNet-18 model that is identical to the one described in Tian et al.
Revised: We use a ResNet-18 model that is identical to the one described in Tian et al.

Original: Following Tao et al.
Revised: Following Tao et al.

Original: (2020b), we use 500 labeled samples per base class to train our feature extractor and 100 for testing.
Revised: (2020b), we use 500 labeled samples per base class to train our feature extractor and 100 for testing.

Original: Incremental evaluation Again following Tao et al.
Revised: Incremental evaluation Again following Tao et al.

Original: (2020b), we evaluate for a total of 8 incremental sessions 1 ≤ t ≤ 8 for mini ImageNet.
Revised: (2020b), we evaluate for a total of 8 incremental sessions 1 ≤ t ≤ 8 for mini ImageNet.

Original: In each session, for S ( t ) , we sample 5 novel classes for training and 5 samples from each class.
Revised: In each session, for S ( t ) , we sample 5 novel classes for training and 5 samples from each class.

Original: Hence, at the last session t = 8 , evaluation involves the entire set of 100 mini ImageNet classes.
Revised: Hence, at the last session t = 8 , evaluation involves the entire set of 100 mini ImageNet classes.

Original: We use GloVe embeddings (Pennington et al., 2014) for label embeddings in Eq.
Revised: We use GloVe embeddings (Pennington et al., 2014) for label embeddings in Eq.

Original: Joint-training In order to assess how our approaches compare to a setting where memory is abundant and retraining is allowed, we train a separate model in each session t from scratch.
Revised: Joint-training In order to assess how our approaches compare to a setting where memory is abundant and retraining is allowed, we train a separate model in each session t from scratch.

Original: These[Table] models are trained on the combination of all training examples including those from the base classes (cid:83) ( t (cid:48) <t ) S
Revised: These models are trained on the combination of all training examples including those from the base classes (cid:83) ( t (cid:48) <t ) S

Original: ( t (cid:48) )
Revised: ( t (cid:48) )

Original: using the loss described in Eq. 2.
Revised: using the loss described in Eq. 2.

Original: Note that in this setup the memory requirement is substantially large (500 examples per each class in C (0) , and 5 for each of the subsequent classes
Revised: Note that in this setup the memory requirement is substantially large (500 examples per each class in C (0) , and 5 for each of the subsequent classes

Original: C (1 ≤ t (cid:48) ≤ t ) ) and the computation is more expensive.
Revised: C (1 ≤ t (cid:48) ≤ t ) ) and the computation is more expensive.

Original: Results Fig.
Revised: Results Fig.

Original: 2 and Table 1 show the results of multi-session experiments with and without memory.
Revised: 2 and Table 1 show the results of multi-session experiments with and without memory.

Original: Session 0 indicates base class accuracy after feature extractor training.
Revised: Session 0 indicates base class accuracy after feature extractor training.

Original: We compare subspace and language-guided regularization (linear mapping and semantic subspace reg.) to simple ﬁnetuning (a surprisingly strong baseline) and joint-training.
Revised: We compare subspace and language-guided regularization (linear mapping and semantic subspace reg.) to simple ﬁnetuning (a surprisingly strong baseline) and joint-training.

Original: We also compare our results to three recent benchmarks: Tao et al.
Revised: We also compare our results to three recent benchmarks: Tao et al.

Original: (2020b), Chen & Lee (2021) and Cheraghian et al.
Revised: (2020b), Chen & Lee (2021) and Cheraghian et al.

Original: When models are evaluated on combined base and novel accuracy, subspace regularization outperforms previous approaches (by 22% (-M) and 7% (+M) at session 8); when semantic information about labels is available, linear mapping and semantic subspace regularization outperform Cheraghian et al.
Revised: When models are evaluated on combined base and novel accuracy, subspace regularization outperforms previous approaches (by 23% (-M) and 10% (+M) at session 8); when semantic information about labels is available, linear mapping and semantic subspace regularization outperform Cheraghian et al.

Original: (Table 1).
Revised: (Table 1).

Original: Evaluating only base sample accuracies (Fig. 2b), semantic subspace reg.
Revised: Evaluating only base sample accuracies (Fig. 2b), semantic subspace reg.

Original: outperforms others; compared to regularization based approaches ﬁne-tuning is subject to catastrophic forgetting.
Revised: outperforms others; compared to regularization based approaches ﬁne-tuning is subject to catastrophic forgetting.

Original: R old is still useful in regulating forgetting (Table 3 in appendix).
Revised: R old is still useful in regulating forgetting (Table 3 in appendix).

Original: The method of Chen & Lee (2021) follows a similar trajectory to our regularizers, but at a much lower accuracy (Fig. 2a).
Revised: The method of Chen & Lee (2021) follows a similar trajectory to our regularizers, but at a much lower accuracy (Fig. 2a).

Original: In Fig.
Revised: In Fig.

Original: 2c, a high degree of forgetting in base classes with ﬁne-tuning allows higher accuracy for novel classes—though not enough to improve average performance (Fig. 2a).
Revised: 2c, a high degree of forgetting in base classes with ﬁne-tuning allows higher accuracy for novel classes—though not enough to improve average performance (Fig. 2a).

Original: By contrast, subspace regularizers enable a good balance between plasticity and stability (Mermillod et al., 2013).
Revised: By contrast, subspace regularizers provide a good balance between plasticity and stability (Mermillod et al., 2013).

Original: In Table 1, storing as few as a single example per an old class substantially helps to reduce forgetting.
Revised: In Table 1, storing as few as a single example per an old class substantially helps to reduce forgetting.

Original: Results from linear mapping and semantic subspace regularization are close, with semantic subspace regularization performing roughly 1% better on average.
Revised: Results from linear mapping and semantic subspace regularization are close, with semantic subspace regularization performing roughly 1% better on average.

Original: The two approaches offer different trade-offs between base and novel accuracies: the latter is more competitive for base classes and vice-versa.
Revised: The two approaches offer different trade-offs between base and novel accuracies: the latter is more competitive for base classes and vice-versa.

Revised: A similar argument can be made when comparing the subspace regularizer with the semantic counterpart where the latter is more efﬁcient in base classes than the former.

Original: Last but not least, keeping as little as one example per old class in the memory along with oursemantic subspace regularizer is sufﬁcient to surpass joint-training—a procedure which is extremely inefﬁcient in terms of both space and computation.
Revised: Last but not least, keeping as little as one example per old class in the memory along with our subspace regularizers is sufﬁcient to surpass joint-training—a procedure which is extremely inefﬁcient in terms of both space and computation.

Original: 5. S INGLE S ESSION
Revised: 5. S INGLE S ESSION

Original: In this section we describe the experiment setup for the single-session evaluation, ( T = 1) , and compare our approach to state-of-the-art XtarNet (Yoon et al., 2020), as well as Ren et al.
Revised: In this section we describe the experiment setup for the single-session evaluation, ( T = 1) , and compare our approach to state-of-the-art XtarNet (Yoon et al., 2020), as well as Ren et al.

Original: (2019), feature extractor substantially

Original: Despite extensive experiments

Original: , 2016), we were unable to identify a training procedure that reproduced the reported accuracy for session 0: all model variants investigated achieved 80%+ validation accuracy.

Original: Gidaris & Komodakis (2018) and Qi et al.
Revised: Gidaris & Komodakis (2018) and Qi et al.

Original: We evaluate our models on 1-shot and 5-shot settings.mini ImageNet and tiered ImageNet For mini ImageNet single-session experiments, we follow the the splits provided by Yoon et al.
Revised: We evaluate our models on 1-shot and 5-shot settings.mini ImageNet and tiered ImageNet For mini ImageNet single-session experiments, we follow the the splits provided by Yoon et al.

Original: Out of 100, 64 classes are used in session t = 0 , 20 in session t = 1 and the remaining for development.
Revised: Out of 100, 64 classes are used in session t = 0 , 20 in session t = 1 and the remaining for development.

Original: Following Yoon et al.
Revised: Following Yoon et al.

Original: , we use ResNet- (a smaller version of the model described in Section 5.1).
Revised: , we use ResNet- (a smaller version of the model described in Section 5.1).

Original: tiered ImageNet (Ren et al., 2018) contains a total of 608 classes out of which 351 are used in session t
Revised: tiered ImageNet (Ren et al., 2018) contains a total of 608 classes out of which 351 are used in session t

Original: = 0 and 160 are reserved for t = 1 .
Revised: = 0 and 160 are reserved for t = 1 .

Original: The remaining 97 are used for development.
Revised: The remaining 97 are used for development.

Original: While previous work (Ren et al., 2019; Yoon et al., 2020) separate 151 classes out of the 351 for meta training, we pool all 351 for feature extractor training.
Revised: While previous work (Ren et al., 2019; Yoon et al., 2020) separate 151 classes out of the 351 for meta training, we pool all 351 for feature extractor training.

Revised: al. (2020b) does not provide an implementation for the main TOPIC algorithm in their released code.

Revised: Therefore, we report published results rather than a reproduction.

Revised: This comparison is inexact: our feature extractor performs substantially better than Tao et al.

Original: We train the same ResNet-18 described in Section 5.1.
Revised: (2020b) and Chen & Lee (2021). Despite extensive experiments (see appendix) on various versions of ResNet-18 (He et al., 2016), we were unable to identify a training procedure that reproduced the reported accuracy for session 0: all model variants investigated achieved 80%+ validation accuracy. Unlike in the preceding section, we were able to successfully reproduce the XtarNet model.

Revised: Our version gives better results on the mini ImageNet dataset but worse results on the tiered ImageNet datasets; for fairness, we thus report results for our version of XtarNet on mini ImageNet and previously reported numbers on tiered ImageNet.

Revised: For other models, we show accuracies reported in previous work.

Revised: [Table]

Revised: We train the same ResNet-18 described in Section 5.1.

Original: Additional details regarding learning rate scheduling, optimizer parameters and other training conﬁgurations may be found in the appendix.
Revised: Additional details regarding learning rate scheduling, optimizer parameters and other training conﬁgurations may be found in the appendix.

Original: Incremental evaluation We follow Yoon et al.
Revised: Incremental evaluation We follow Yoon et al.

Original: (2020) for evaluation.
Revised: (2020) for evaluation.

Original: Yoon et al.
Revised: Yoon et al.

Original: independently sample 2000 D (1) incremental datasets (“episodes”) from the testing classes C (1) .
Revised: independently sample 2000 D (1) incremental datasets (“episodes”) from the testing classes C (1) .

Original: They report average accuracies over all episodes with 95% conﬁdence intervals.
Revised: They report average accuracies over all episodes with 95% conﬁdence intervals.

Original: At every episode, Q (1) is resampled from both base and novel classes, C (0) and C (1) , with equal probability for both mini ImageNet and tiered ImageNet.
Revised: At every episode, Q (1) is resampled from both base and novel classes, C (0) and C (1) , with equal probability for both mini ImageNet and tiered ImageNet.

Original: We again ﬁne-tune the weights until convergence.
Revised: We again ﬁne-tune the weights until convergence.

Original: We do not reserve samples from base classes, thus the only training samples during incremental evaluation is from the novel classes
Revised: We do not reserve samples from base classes, thus the only training samples during incremental evaluation is from the novel classes

Original: C (1) .
Revised: C (1) .

Original: We use the same resources for label embeddings and Sentence-BERT embeddings (Reimers & Gurevych, 2019) for descriptions which are retrieved from WordNet (Miller, 1995).
Revised: We use the same resources for label embeddings and Sentence-BERT embeddings (Reimers & Gurevych, 2019) for descriptions which are retrieved from WordNet (Miller, 1995).

Original: Results We report aggregate results for 1-shot and 5-shots settings of mini ImageNet and tiered ImageNet (Table 2).
Revised: Results We report aggregate results for 1-shot and 5-shots settings of mini ImageNet and tiered ImageNet (Table 2).

Original: Compared to previous work specialized for the single-session setup without a straightforward way to expand into multi-session (Ren et al., 2019; Yoon et al., 2020), even our simple ﬁne-tuning baseline perform well on both datasets—outperforming the previous stateof-the-art in three out of four settings in Table 2.
Revised: Compared to previous work specialized for the single-session setup without a straightforward way to expand into multi-session (Ren et al., 2019; Yoon et al., 2020), even our simple ﬁne-tuning baseline perform well on both datasets—outperforming the previous stateof-the-art in three out of four settings in Table 2.

Original: Addition of subspace and semantic regularization improves performance overall but tiered ImageNet 1-shot setting.
Revised: Addition of subspace and semantic regularization improves performance overall but tiered ImageNet 1-shot setting.

Original: Semantic subspace regularizers match or outperform linear label mapping.
Revised: Semantic subspace regularizers match or outperform linear label mapping.

Original: Subspace regularization outperforms ﬁne-tuning in 5shot settings and matches it in 1-shot.
Revised: Subspace regularization outperforms ﬁne-tuning in 5shot settings and matches it in 1-shot.

Original: In addition to accuracy, we report a quantity labeled ∆ by
Revised: In addition to accuracy, we report a quantity labeled ∆ by

Original: Ren et al.
Revised: Ren et al.

Original: ∆ serves as a measure of catastrophic forgetting, with the caveat that it can be minimized by a model that achieves a classiﬁcation accuracy of 0 on both base and novel classes.
Revised: ∆ serves as a measure of catastrophic forgetting, with the caveat that it can be minimized by a model that achieves a classiﬁcation accuracy of 0 on both base and novel classes.

Original: We ﬁnd that our approaches result in approximately the same ∆ in mini ImageNet and worse in tiered ImageNet than previous work.
Revised: We ﬁnd that our approaches result in approximately the same ∆ in mini ImageNet and worse in tiered ImageNet than previous work.

Original:  A NALYSIS AND L IMITATIONS
Revised:  A NALYSIS AND L IMITATIONS

Original: What does regularization actually do?
Revised: What does regularization actually do?

Original: Fine-tuning results in prediction biased towards the most recently learned classes (top of Fig.
Revised: Fine-tuning results in prediction biased towards the most recently learned classes (top of Fig.

Original: 3) when no subspace regularization is imposed.
Revised: 3) when no subspace regularization is imposed.

Original: Our experiments show that preserving the base weights while regularizing novel weights gives signiﬁcant improvements over ordinary ﬁne-tuning (bottom of Fig.
Revised: Our experiments show that preserving the base weights while regularizing novel weights gives signiﬁcant improvements over ordinary ﬁne-tuning (bottom of Fig.

Original: 3)—resulting a fairer prediction over all classes and reducing catastrophic forgetting.
Revised: 3)—resulting a fairer prediction over all classes and reducing catastrophic forgetting.

Original: In Table 1, Semantic Subspace Reg.
Revised: In Table 1, Semantic Subspace Reg.

Original: results in 73.8% and 47.6% accuracies in the 1 st and 8 th sessions whereas, ﬁne-tuning results in 61.8% and 26.5%, respectively, even without any memory—suggesting that regularization ensures a better retention of accuracy.
Revised: results in 73.8% and 47.6% accuracies in the 1 st and 8 th sessions whereas, ﬁne-tuning results in 61.8% and 26.5%, respectively, even without any memory—suggesting that regularization ensures a better retention of accuracy.

Original: While the trade-off between accuracies of base and novel classes is inevitable due to the nature of the classiﬁcation problem, the proposed regularizers provide a good balance between the two.
Revised: While the trade-off between accuracies of base and novel classes is inevitable due to the nature of the classiﬁcation problem, the proposed regularizers provide a good balance between the two.

Original: What are the limitations of the proposed regularization scheme?
Revised: What are the limitations of the proposed regularization scheme?

Original: Our approach targets only errors that originate in the ﬁnal layer of the model—while a convolutional feature extractor is used, the parameters of this feature extractor are ﬁxed, and we have focused on FSCIL as a linear classiﬁcation problem.
Revised: Our approach targets only errors that originate in the ﬁnal layer of the model—while a convolutional feature extractor is used, the parameters of this feature extractor are ﬁxed, and we have focused on FSCIL as a linear clas- siﬁcation problem.

Original: Future work might extend these approaches to incorporate ﬁne-tuning of the (nonlinear) feature extractor itself while preserving performance on all classes in the longer term.
Revised: Future work might extend these approaches to incorporate ﬁne-tuning of the (nonlinear) feature extractor itself while preserving performance on all classes in the longer term.

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: We have described a family of regularization-based approaches to few-shot class-incremental learning, drawing connections between incremental learning and the general multi-task and zero-shot learning literature.
Revised: We have described a family of regularization-based approaches to few-shot class-incremental learning, drawing connections between incremental learning and the general multi-task and zero-shot learning literature.

Original: The proposed regularizers are extremely simple—they involve only one extra hyperparameter, require no additional training steps or model parameters, and are easy to understand and implement.
Revised: The proposed regularizers are extremely simple—they involve only one extra hyperparameter, require no additional training steps or model parameters, and are easy to understand and implement.

Original: Despite this simplicity, our approach enables ordinary classiﬁcation architectures to achieve state-of-the-art results on the doubly challenging few-shot incremental image classiﬁcation across multiple datasets and problem formulations.
Revised: Despite this simplicity, our approach enables ordinary classiﬁcation architectures to achieve state-of-the-art results on the doubly challenging few-shot incremental image classiﬁcation across multiple datasets and problem formulations.

Original: A CKNOWLEDGMENTS
Revised: A CKNOWLEDGMENTS

Original: This work is supported in part by the U S. NSF grant 1838193 and DARPA HR001118S0044 (the
Revised: This work is supported in part by the U.S. NSF grant 1838193 and DARPA HR001118S0044 (the

Original: LwLL program).
Revised: LwLL program).

Original: The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes.
Revised: The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes.

Original: The views and conclusions contained in this publication are those of the authors and should not be interpreted as representing ofﬁcial policies or endorsements of DARPA and
Revised: The views and conclusions contained in this publication are those of the authors and should not be interpreted as representing ofﬁcial policies or endorsements of DARPA and

Original: the
Revised: the

Original: U S. Government.
Revised: U.S. Government.

Original: Computing resources were provided by the MIT Supercloud (Reuther et al., 2018).
Revised: Computing resources were provided by the MIT Supercloud (Reuther et al., 2018).

Original: A C ODE AND D ATASETS
Revised: A C ODE AND D ATASETS

Original: Code will be made publicly available.
Revised: Code is released under https://github.com/feyzaakyurek/subspace-reg .

Original: We use mini ImageNet (Vinyals et al., 2016) and tiered ImageNet (Ren et al., 2018)
Revised: We use mini ImageNet (Vinyals et al., 2016) and tiered ImageNet (Ren et al., 2018)

Original: datasets both are subsets of ImageNet dataset (Russakovsky et al., 2015).
Revised: datasets both are subsets of ImageNet dataset (Russakovsky et al., 2015).

Original: Use of terms and licenses are available through the respective sources.
Revised: Use of terms and licenses are available through the respective sources.

Original: B A NALYSIS OF C ATASTROPHIC F ORGETTING
Revised: B A NALYSIS OF C ATASTROPHIC F ORGETTING

Original: In Table
Revised: In Table

Original: [Table]
Revised: [Table]

Original: we demonstrate the effectiveness of the regularization term R old in mitigating catastrophic forgetting.
Revised: we demonstrate the effectiveness of the regularization term R old in mitigating catastrophic forgetting.

Original: usefulness

Original: R old reducing catastrophic forgetting multiple

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: C R ESULTS IN T ABULAR F ORM
Revised: C R ESULTS IN T ABULAR F ORM

Original: In Table 4 and Table 5, we present the multi-session results in the main paper in the tabular form.
Revised: In Table 4 and Table 5, we present the multi-session results in the main paper in the tabular form.

Original: D D ETAILS OF F EATURE E XTRACTOR T RAINING
Revised: D D ETAILS OF F EATURE E XTRACTOR T RAINING

Original: We use the exact ResNet described in Tian et al.
Revised: We use the exact ResNet described in Tian et al.

Original: (2020), the differences compared to the standard
Revised: (2020), the differences compared to the standard

Original: ResNet (He et al., 2016): (1) Each block (collection of convolutional blocks) is composed of three convolutional layers instead of two.
Revised: ResNet (He et al., 2016): (1) Each block (collection of convolutional blocks) is composed of three convolutional layers instead of two.

Original: (2) Number of blocks for ResNet-12 is 4 instead of 6 of the standard version, thus the total number of convolutional layers are the same.
Revised: (2) Number of blocks for ResNet-12 is 4 instead of 6 of the standard version, thus the total number of convolutional layers are the same.

Original: (3) Filter sizes are [64,160,320,640] rather than [64,128,256,512], though the total number of ﬁlters is comparable since Tian et al.
Revised: (3) Filter sizes are [64,160,320,640] rather than [64,128,256,512], though the total number of ﬁlters is comparable since Tian et al.

Original: (2020) has less blocks.
Revised: (2020) has less blocks.

Original: (4) There is Dropblock at the end of the last blocks.
Revised: (4) There is Dropblock at the end of the last blocks.

Original: Tian et al.
Revised: Tian et al.

Original: (2020) provides a full visualization in Appendix and their code repository 6 is easy to browse on which we base our own codebase.
Revised: (2020) provides a full visualization in Appendix and their code repository 6 is easy to browse on which we base our own codebase.

Original: We observe that the previous work oftentimes use their slightly modiﬁed version of the standard ResNet.
Revised: We observe that the previous work oftentimes use their slightly modiﬁed version of the standard ResNet.

Original: Ren et al.
Revised: Ren et al.

Original: (2019) uses the ResNet-10 (Mishra et al., 2017) and ResNet-18 for for mini ImageNet and tiered ImageNet, respectively.
Revised: (2019) uses the ResNet-10 (Mishra et al., 2017) and ResNet-18 for for mini ImageNet and tiered ImageNet, respectively.

Original: XtarNet(Yoon et al., 2020) is originally based on a slightly modiﬁed version of ResNet-12 and ResNet-18 which we replaced with our version, improving their results for mini ImageNet but not in tiered ImageNet, thus we report improved results for mini ImageNet and their results for tiered ImageNet in the main paper.
Revised: XtarNet(Yoon et al., 2020) is originally based on a slightly modiﬁed version of ResNet-12 and ResNet-18 which we replaced with our version, improving their results for mini ImageNet but not in tiered ImageNet, thus we report improved results for mini ImageNet and their results for tiered ImageNet in the main paper.

Original: [Table]
Revised: [Table]

Original: D. D EFAULT S ETTINGS
Revised: D. D EFAULT S ETTINGS

Original: Unless otherwise indicated we use the following default settings of Tian et al.
Revised: Unless otherwise indicated we use the following default settings of Tian et al.

Original: (2020) in our feature extractor training.
Revised: (2020) in our feature extractor training.

Original: We use SGD optimizer with learning starting at 0.05 with decays by 0.1 at epochs 60 and 80.
Revised: We use SGD optimizer with learning starting at 0.05 with decays by 0.1 at epochs 60 and 80.

Original: We train for a total of 100 epochs.
Revised: We train for a total of 100 epochs.

Original: Weight decay is 5e-4, momentum is 0.9 and batch size is 64.
Revised: Weight decay is 5e-4, momentum is 0.9 and batch size is 64.

Original: As per transformations on training images, we use random crop of 84x84 with padding 8.
Revised: As per transformations on training images, we use random crop of 84x84 with padding 8.

Original: We also use color jitter ( brightness =0.4, contrast =0.4, saturation =0.4) and horizontal ﬂips.
Revised: We also use color jitter ( brightness =0.4, contrast =0.4, saturation =0.4) and horizontal ﬂips.

Original: For each run, we sample 1000 images from base classes and 25 images from each of novel classes.
Revised: For each run, we sample 1000 images from base classes and 25 images from each of novel classes.

Original: Our classiﬁer does not have bias.
Revised: Our classiﬁer does not have bias.

Original: D. M ULTI -S ESSION mini I MAGE N ET
Revised: D. M ULTI -S ESSION mini I MAGE N ET

Original: We re-sample the set of base classes (60 classes) 10 times across different seeds and train ten ResNet18 models.
Revised: We re-sample the set of base classes (60 classes) 10 times across different seeds and train ten ResNet18 models.

Original: Each class has 500 training images.
Revised: Each class has 500 training images.

Original: We follow the default settings for training.
Revised: We follow the default settings for training.

Original: D. M ULTI -S ESSION C OMPARISON TO S TANDARD R ES N ET - 
Revised: D. M ULTI -S ESSION C OMPARISON TO S TANDARD R ES N ET -

Original: In Table 6 we provide validation set results for two types of ResNet-18’s: Tian et al.
Revised: In Table 6 we provide validation set results for two types of ResNet-18’s: Tian et al.

Original: and He et al.
Revised: and He et al.

Original: across ten different seeds.
Revised: across ten different seeds.

Original: Results show that use of Tian et al.
Revised: Results show that use of Tian et al.

Original: (2020) does not incur unfair advantage over those who used He et al.
Revised: (2020) does not incur unfair advantage over those who used He et al.

Original: D. S INGLE -S ESSION mini I MAGE N ET
Revised: D. S INGLE -S ESSION mini I MAGE N ET

Original: We follow the default hyperparameters parameters Appendix D.1 except that for training, validation and testing we use the exact splits provided by Ren et al.
Revised: We follow the default hyperparameters parameters Appendix D.1 except that for training, validation and testing we use the exact splits provided by Ren et al.

Original: (2019) also used by Yoon et al.
Revised: (2019) also used by Yoon et al.

Original: There are 64 base, 16 validation and 20 testing classes provided (totaling 100).
Revised: There are 64 base, 16 validation and 20 testing classes provided (totaling 100).

Original: Training data consists of 600 images per base class.
Revised: Training data consists of 600 images per base class.

Original: Dataset statistics are delineated in the Appendix of Ren et al.
Revised: Dataset statistics are delineated in the Appendix of Ren et al.

Original: and downloadable splits are available here, courtesy of Ren et al.
Revised: and downloadable splits are available here, courtesy of Ren et al.

Original: D. S INGLE -S ESSION tiered I MAGE N ET
Revised: D. S INGLE -S ESSION tiered I MAGE N ET

Original: tiered ImageNet is ﬁrst introduced by Ren et al.
Revised: tiered ImageNet is ﬁrst introduced by Ren et al.

Original: Same as above, we use the default parameters except that we train for a total of 60 epochs decaying the initial learning rate of 0.05 by 0.1 at epochs 30 and 45.
Revised: Same as above, we use the default parameters except that we train for a total of 60 epochs decaying the initial learning rate of 0.05 by 0.1 at epochs 30 and 45.

Original: Again, we use the same data as previous work available at the same link above.
Revised: Again, we use the same data as previous work available at the same link above.

Original: tiered ImageNet is split into 351, 97 and 160 classes.
Revised: tiered ImageNet is split into 351, 97 and 160 classes.

Original: Past work that use meta-learning Ren et al.
Revised: Past work that use meta-learning Ren et al.

Original: ; Yoon et al.
Revised: ; Yoon et al.

Original: (2020) split 351 training classes into further 200 and 151 clases where the latter is used for meta learning.
Revised: (2020) split 351 training classes into further 200 and 151 clases where the latter is used for meta learning.

Original: We pool all 351 for feature extractor training.
Revised: We pool all 351 for feature extractor training.

Original: At the end of feature extractor training, we only keep the classiﬁer weights for the ﬁrst 200 classes to adhere to the evaluation scheme of 200+5 classes as past work.
Revised: At the end of feature extractor training, we only keep the classiﬁer weights for the ﬁrst 200 classes to adhere to the evaluation scheme of 200+5 classes as past work.

Original: E D ETAILS OF I NCREMENTAL E VALUATION E. QR D ECOMPOSITION FOR S UBSPACE R EGULARIZATION
Revised: E D ETAILS OF I NCREMENTAL E VALUATION E. QR D ECOMPOSITION FOR S UBSPACE R EGULARIZATION

Original: To compute the orthogonal basis P C (0) for the subspace spanned by base classiﬁer weights η C (0) we use QR decomposition(Trefethen & Bau III, 1997):
Revised: To compute the orthogonal basis P C (0) for the subspace spanned by base classiﬁer weights η C (0) we use QR decomposition(Trefethen & Bau III, 1997):

Original: [Equation]
Revised: [Equation]

Original: E. M ULTI -S ESSION
Revised: E. M ULTI -S ESSION

Original: For testing, we sample 1000 images from base classes and 25 images from each of novel classes.
Revised: For testing, we sample 1000 images from base classes and 25 images from each of novel classes.

Original: Testing images from a given class stay the same across sessions.
Revised: Testing images from a given class stay the same across sessions.

Original: Harmonic mean results take into account the ratio of base classes to novel classes in a given session.
Revised: Harmonic mean results take into account the ratio of base classes to novel classes in a given session.

Original: In this setting, there is no explicit development set (with disjoint classes than train and test) deﬁned by previous work thus we use the ﬁrst incremental learning session (containing 5 novel classes) as our development set.
Revised: In this setting, there is no explicit development set (with disjoint classes than train and test) deﬁned by previous work thus we use the ﬁrst incremental learning session (containing 5 novel classes) as our development set.

Original: Default settings We use the same transformations as in Appendix D.1 on the training images.
Revised: Default settings We use the same transformations as in Appendix D.1 on the training images.

Original: We stop ﬁne-tuning when loss does not change more than 0.0001 for at least 10 epochs.
Revised: We stop ﬁne-tuning when loss does not change more than 0.0001 for at least 10 epochs.

Original: We use
Revised: We use

Original: SGD optimizer.
Revised: SGD optimizer.

Original: We repeat the experiments 10 times and report the average accuracy with standard deviation (95% conﬁdence interval) in the paper.
Revised: We repeat the experiments 10 times and report the average accuracy with standard deviation (95% conﬁdence interval) in the paper.

Original: Simple Fine-tuning We use learning rate of 0.002 and do not use learning decay.
Revised: Simple Fine-tuning We use learning rate of 0.002 and do not use learning decay.

Original: Weight-decay α is set at 5e-3.
Revised: Weight-decay α is set at 5e-3.

Original: In order to limit the change in weights, we use different β ’s for base and previously learned novel classes, where the former is 0.2 and the latter 0.1.
Revised: In order to limit the change in weights, we use different β ’s for base and previously learned novel classes, where the former is 0.2 and the latter 0.1.

Original: We rely on the default settings otherwise.
Revised: We rely on the default settings otherwise.

Original: Subspace Regularization Different from simple ﬁne-tuning we use a weight decay of 5e-4.
Revised: Subspace Regularization Different from simple ﬁne-tuning we use a weight decay of 5e-4.

Original: There is an additional parameter called γ in this setting controlling the degree of pulling of novel weights toward the subspace which we set to 1.0.
Revised: There is an additional parameter called γ in this setting controlling the degree of pulling of novel weights toward the subspace which we set to 1.0.

Original: Semantic Subspace Regularization Different than simple subspace regularization, there is a temperature parameter used in the Softmax operation used in computation of l c ’s which we set to 3.0.
Revised: Semantic Subspace Regularization Different than simple subspace regularization, there is a temperature parameter used in the Softmax operation used in computation of l c ’s which we set to 3.0.

Original: Linear mapping regularization Same parameters as in subspace regularization are used except γ = 0 .
Revised: Linear mapping regularization Same parameters as in subspace regularization are used except γ = 0 .

Original: We formulate L as a linear layer with bias and we use gradient descent to train the parameters.
Revised: We formulate L as a linear layer with bias and we use gradient descent to train the parameters.

Original: E. S INGLE -S ESSION
Revised: E. S INGLE -S ESSION

Original: In our 1-shot experiments unless ﬁne-tuning converges by then, we stop at the maximum number of epochs at 1000.
Revised: In our 1-shot experiments unless ﬁne-tuning converges by then, we stop at the maximum number of epochs at 1000.

Original: We sample 2000 episodes which includes 5 novels classes and 1-5 samples from each and report average accuracy.
Revised: We sample 2000 episodes which includes 5 novels classes and 1-5 samples from each and report average accuracy.

Original: For testing, base and novel samples have equal weight in average per previous work (Yoon et al., 2020).
Revised: For testing, base and novel samples have equal weight in average per previous work (Yoon et al., 2020).

Original: SGD optimizer is used.
Revised: SGD optimizer is used.

Original: For description similarity we use
Revised: For description similarity we use

Original: Sentence-BERT’s stsb-roberta-large .
Revised: Sentence-BERT’s stsb-roberta-large .

Original: mini ImageNet Settings We use the same set of transformations on the training images as described in Appendix D.1.
Revised: mini ImageNet Settings We use the same set of transformations on the training images as described in Appendix D.1.

Original: We ﬁrst describe details of 1-shot setting.
Revised: We ﬁrst describe details of 1-shot setting.

Original: In 1-shot experiments we set the maximum epochs to 1000.
Revised: In 1-shot experiments we set the maximum epochs to 1000.

Original: For simple ﬁne-tuning we use learning rate of 0.003, and weight decay of 5e-3.
Revised: For simple ﬁne-tuning we use learning rate of 0.003, and weight decay of 5e-3.

Original: In Semantic Subspace Reg.
Revised: In Semantic Subspace Reg.

Original: , we set temperature to 1.5.
Revised: , we set temperature to 1.5.

Original: Both in Semantic Subspace Reg.
Revised: Both in Semantic Subspace Reg.

Original: and linear mapping γ = 0 .
Revised: and linear mapping γ = 0 .

Original: 005 and weight-decay is 5e-4.
Revised: 005 and weight-decay is 5e-4.

Original: In subspace regularization, γ = 0 .
Revised: In subspace regularization, γ = 0 .

Original: 005 and weight-decay is set to 5e-5.
Revised: 005 and weight-decay is set to 5e-5.

Original: Description similarity follows the same setup as Semantic Subspace
Revised: Description similarity follows the same setup as Semantic Subspace

Original: Reg..
Revised: Reg..

Original: In 5-shot setting, we set β = 0 .
Revised: In 5-shot setting, we set β = 0 .

Original: 03 weight-decay to 5e-3 and learning rate to 0.002.
Revised: 03 weight-decay to 5e-3 and learning rate to 0.002.

Original: For subspace regularization, Semantic Subspace Reg.
Revised: For subspace regularization, Semantic Subspace Reg.

Original: and linear mapping we use γ = 0 .
Revised: and linear mapping we use γ = 0 .

Original: 03 and for description similarity we use γ = 0 .
Revised: 03 and for description similarity we use γ = 0 .

Original: tiered ImageNet Settings
Revised: tiered ImageNet Settings

Original: In 1-shot setting, ﬁne-tuning uses learning rate of 0.003.
Revised: In 1-shot setting, ﬁne-tuning uses learning rate of 0.003.

Original: Semantic Subspace Reg.
Revised: Semantic Subspace Reg.

Original: has learning rate of 0.005, weight-decay 5e-3 and γ = 0 .
Revised: has learning rate of 0.005, weight-decay 5e-3 and γ = 0 .

Original: Subspace reg.
Revised: Subspace reg.

Original: and linear mapping use γ = 0 .
Revised: and linear mapping use γ = 0 .

Original: In 5-shot setting, for simple ﬁne-tuning we set lr = 0.001, weight-decay=5e-3, β = 0 .
Revised: In 5-shot setting, for simple ﬁne-tuning we set lr = 0.001, weight-decay=5e-3, β = 0 .

Original: For Semantic Subspace Reg.
Revised: For Semantic Subspace Reg.

Original: γ = 0 .
Revised: γ = 0 .

Original: 05 and β = 0 .
Revised: 05 and β = 0 .

Original: 2 while others have γ = 0 .
Revised: 2 while others have γ = 0 .

Original: F V ISUALIZATIONS
Revised: F V ISUALIZATIONS

Original: In Fig.
Revised: In Fig.

Original: 4 and Fig.
Revised: 4 and Fig.

Original: 5 we depict principal components of classiﬁer weights as well as semantic or subspace targets for novel weights.
Revised: 5 we depict principal components of classiﬁer weights as well as semantic or subspace targets for novel weights.

Original: G C OMPUTE
Revised: G C OMPUTE

Original: We use a single 32 GB V100 NVIDIA
Revised: We use a single 32 GB V100 NVIDIA

Original: GPU for all our experiments.
Revised: GPU for all our experiments.