Original: ALIFE: Adaptive Logit Regularizer and Feature Replay for Incremental Semantic Segmentation
Revised: ALIFE: Adaptive Logit Regularizer and Feature Replay for Incremental Semantic Segmentation

Original: Anonymous Author(s)

Original: AfﬁliationAddress email

Original: Abstract
Revised: Abstract

Original: We address the problem of incremental semantic segmentation (ISS) recognizingnovel object/stuff categories continually without forgetting previous ones that havebeen learned.
Revised: We address the problem of incremental semantic segmentation (ISS) recognizing novel object/stuff categories continually without forgetting previous ones that have been learned.

Original: The catastrophic forgetting problem is particularly severe in ISS,since pixel-level ground-truth labels are available only for the novel categoriesat training time.
Revised: The catastrophic forgetting problem is particularly severe in ISS, since pixel-level ground-truth labels are available only for the novel categories at training time.

Original: To address the problem, regularization-based methods exploitprobability calibration techniques to learn semantic information from unlabeledpixels.
Revised: To address the problem, regularization-based methods exploit probability calibration techniques to learn semantic information from unlabeled pixels.

Original: While such techniques are effective, there is still  lack of theoretical support.
Revised: While such techniques are effective, there is still a lack of theoretical understanding of them.

Original: Replay-based methods propose to memorize a small set of images for previouscategories.
Revised: Replay-based methods propose to memorize a small set of images for previous categories.

Original: They achieve state-of-the-art performance at the cost of large memoryfootprint.
Revised: They achieve state-of-the-art performance at the cost of large memory footprint.

Original: We propose in this paper a novel ISS method, dubbed ALIFE, thatprovides a better compromise between accuracy and efﬁciency.
Revised: We propose in this paper a novel ISS method, dubbed ALIFE, that provides a better compromise between accuracy and efﬁciency.

Original: To this end, we ﬁrstshow an in-depth analysis on the calibration techniques to better understand theeffects on ISS.
Revised: To this end, we ﬁrst show an in-depth analysis on the calibration techniques to better understand the effects on ISS.

Original: Based on this, we then introduce an adaptive logit regularizer (ALI)that enables our model to better learn new categories, while retaining knowledgefor previous ones.
Revised: Based on this, we then introduce an adaptive logit regularizer (ALI) that enables our model to better learn new categories, while retaining knowledge for previous ones.

Original: We also present a feature replay scheme that memorizes features,instead of images directly, in order to reduce memory requirements signiﬁcantly.
Revised: We also present a feature replay scheme that memorizes features, instead of images directly, in order to reduce memory requirements signiﬁcantly.

Original: Since a feature extractor is changed continually, memorized features should alsobe updated at every incremental stage.
Revised: Since a feature extractor is changed continually, memorized features should also be updated at every incremental stage.

Original: To handle this, we introduce categoryspeciﬁc rotation matrices updating the features for each category separately.
Revised: To handle this, we introduce category-speciﬁc rotation matrices updating the features for each category separately.

Original: Wedemonstrate the effectiveness of our approach with extensive experiments onstandard ISS benchmarks, and show that our method achieves a better trade-offin terms of accuracy and efﬁciency.
Revised: We demonstrate the effectiveness of our approach with extensive experiments on standard ISS benchmarks, and show that our method achieves a better trade-off in terms of accuracy and efﬁciency.

Original: Our code and model will be made publiclyavailable online.

Original:  Introduction
Revised:  Introduction

Original: Humans are capable of learning new concepts continually, while preserving or even improvingpreviously acquired knowledge.
Revised: Humans are capable of learning new concepts continually, while preserving or even improving previously acquired knowledge.

Original: Artiﬁcial neural networks are, however, prone to forget the knowledgethey have learned if being trained with samples for new object/scene categories alone.
Revised: Artiﬁcial neural networks are, however, prone to forget the knowledge they have learned if being trained with samples for new object/scene categories alone.

Original: The reasonfor this problem, so-called catastrophic forgetting [11, 27], is that parameters of neural networkschange abruptly to handle new categories without accessing training samples for previous categories.
Revised: The reason for this problem, so-called catastrophic forgetting [11, 27], is that parameters of neural networks change abruptly to handle new categories without accessing training samples for previous categories.

Original: A straightforward way to alleviate the problem is to re-train a model with training examples for entirecategories observed so far, which is however computationally demanding.
Revised: A straightforward way to alleviate the problem is to re-train a model with training examples for entire categories observed so far, which is however computationally demanding.

Original: Incremental learning is an alternative approach to learning new categories continuously withoutre-training on the entire dataset.
Revised: Incremental learning is an alternative approach to learning new categories continuously without re-training on the entire dataset.

Original: While many methods have been proposed for incremental classiﬁcation [4, 21, 23, 25, 33, 36], a few attempts explore incremental semantic segmentation (ISS), wheretraining images for new categories are partially labeled to reduce the cost for manual annotation.
Revised: While many methods have been proposed for incremental classiﬁcation [4, 21, 23, 25, 33, 36], a few attempts explore incremental semantic segmentation (ISS), where training images for new categories are partially labeled to reduce the cost for manual annotation.

Original: Thatis, pixels for new categories are labeled only, while remaining ones are marked as unknown.
Revised: That is, pixels for new categories are labeled only, while remaining ones are marked as unknown.

Original: Theunknown regions should be considered separately, since they could contain previous categories alongwith ones that would be seen in the future.
Revised: The unknown regions should be considered separately, since they could contain previous categories along with ones that would be seen in the future.

Original: [Figure]
Revised: [Figure]

Original: MiBSDRPLOPSSULSSUL
Revised: MiBSDRPLOPSSULSSUL

Original: [Figure]
Revised: [Figure]

Revised: MiBSDRPLOPSSULSSUL

Revised: [Figure]

Original:  Current ISS methods can be categorized into regularization-based [2, 9, 30] and replay-based approaches [3, 26].
Revised: M RECALLOursOurs [Figure] Current ISS methods can be categorized into regularization-based [2, 9, 30] and replay-based approaches [3, 26].

Original: Regularization-based methods typically exploit knowledge distillation (KD) [15] topreserve the discriminative ability of ISS models for previous categories.
Revised: Regularization-based methods typically exploit knowledge distillation (KD) [15] to preserve the discriminative ability of ISS models for previous categories.

Original: In particular, the seminalwork of [2] alleviates the semantic shift of unknown regions.
Revised: In particular, the seminal work of [2] alleviates the semantic shift of unknown regions.

Original: Speciﬁcally, it introduces calibratedcross-entropy (CCE) and calibrated KD (CKD) terms to learn semantic information from unknownregions.
Revised: Speciﬁcally, it introduces calibrated cross-entropy (CCE) and calibrated KD (CKD) terms to learn semantic information from unknown regions.

Original: Although CCE and CKD terms have shown the effectiveness on ISS methods [2, 9, 30], thereis still  lack of theoretical support.
Revised: Although CCE and CKD terms have shown the effectiveness on ISS methods [2, 9, 30], there is still a lack of theoretical understanding.

Original: Replay-based methods propose to use a replay buffer consistingof web-crawled or previously seen images.
Revised: Replay-based methods propose to use a replay buffer consisting of web-crawled or previously seen images.

Original: The replay buffer provides rich information storing theknowledge for previous categories, but it incurs large memory footprint, particularly for the task ofsemantic segmentation that typically adopts high-resolution images.
Revised: The replay buffer provides rich information storing the knowledge for previous categories, but it incurs large memory footprint, particularly for the task of semantic segmentation that typically adopts high-resolution images.

Original: In this paper, we present a novel ISS method, dubbed ALIFE, that 1) alleviates catastrophic forgettingand 2) reduces memory requirements.
Revised: In this paper, we present a novel ISS method, dubbed ALIFE, that 1) alleviates catastrophic forgetting and 2) reduces memory requirements.

Original: For the ﬁrst aspect, we analyze gradients of CCE and CKDterms for better understanding the effectiveness on catastrophic forgetting, and make the followingobservations: (1) For unknown regions, CCE reduces logit values of new categories, which is crucialfor preventing overﬁtting to the new categories.
Revised: For the ﬁrst aspect, we analyze gradients of CCE and CKD terms for better understanding the effectiveness on catastrophic forgetting, and make the following observations: (1) For unknown regions, CCE reduces logit values of new categories, which is crucial for preventing overﬁtting to the new categories.

Original: However, it always raises logit values of all previouscategories, without considering whether predictions for those regions are correct or not, which lessensthe discriminative power for previous categories.
Revised: However, it always raises logit values of all previous categories, without considering whether predictions for those regions are correct or not, which lessens the discriminative power for previous categories.

Original: (2) CKD makes it difﬁcult to distinguish newcategories from a background one.
Revised: (2) CKD makes it difﬁcult to distinguish new categories from a background one.

Original: Motivated by these observations, we introduce an adaptive logitregularizer (ALI) that enables better learning new categories and alleviating catastrophic forgetting forprevious ones (Fig. 1(a)).
Revised: Motivated by these observations, we introduce an adaptive logit regularizer (ALI) that enables better learning new categories and alleviating catastrophic forgetting for previous ones (Fig. 1(a)).

Original: For the second aspect, we propose to exploit latent features for replaying,reducing memory footprint and avoiding data privacy issues ( e.g. , in medical imaging [35]).
Revised: For the second aspect, we propose to exploit latent features for replaying, reducing memory footprint and avoiding data privacy issues ( e.g. , in medical imaging [35]).

Original: Incontrast to the replay buffer storing images directly, our approach to using the features updatesthem at every incremental stage, as a feature extractor is also updated continually at training time.
Revised: In contrast to the replay buffer storing images directly, our approach to using the features updates them at every incremental stage, as a feature extractor is also updated continually at training time.

Original: Speciﬁcally, we exploit category-speciﬁc rotation matrices using the Cayley transform.
Revised: Speciﬁcally, we exploit category-speciﬁc rotation matrices using the Cayley transform.

Original: Rotatinglatent features is computationally efﬁcient, while maintaining the relations between the features.
Revised: Rotating latent features is computationally efﬁcient, while maintaining the relations between the features.

Original: Wedemonstrate that exploiting ALI with memorizing features achieves a better trade-off in terms ofaccuracy and efﬁciency (Fig. 1(b)).
Revised: We demonstrate that exploiting ALI with memorizing features achieves a better trade-off in terms of accuracy and efﬁciency (Fig. 1(b)).

Original: Our main contributions are summarized as follows:
Revised: Our main contributions are summarized as follows:

Original: • We show an in-depth analysis of probability calibration methods widely used for ISS [2, 9, 30],and introduce ALI that enables our model to better learn new categories, while maintaining theknowledge for previous categories.•
Revised: • We show an in-depth analysis of probability calibration methods widely used for ISS [2, 9, 30], and introduce ALI that enables our model to better learn new categories, while maintaining the knowledge for previous categories.

Original:  We present a novel replay strategy using category-speciﬁc rotation matrices, which helps toalleviate catastrophic forgetting for previous categories with much less memory requirementsthan replaying raw images.• Extensive experiments demonstrate the effectiveness of our approach to using ALI and replayingfeatures with rotation matrices.
Revised: • We present a novel replay strategy using category-speciﬁc rotation matrices, which helps to alleviate catastrophic forgetting for previous categories with much less memory requirements than replaying raw images. • Extensive experiments demonstrate the effectiveness of our approach to using ALI and replaying features with rotation matrices.

Original: We set a new state of the art on standard ISS benchmarks [10, 39].
Revised: We set a new state of the art on standard ISS benchmarks [10, 40].

Original:  Related work
Revised:  Related work

Original: Many incremental learning approaches have been introducedtrying to preserve knowledge forprevious categories, while recognizing new ones [28].
Revised: Many incremental learning approaches have been introduced to preserve knowledge for previous categories, while recognizing new ones [28].

Original: They can be categorized into task-incremental and class-incremental methods.
Revised: They can be categorized into task-incremental and classincremental methods.

Original: Task-incremental methods [21, 23, 25] treat learning a new setof categories as a new task.
Revised: Task-incremental methods [21, 23, 25] treat learning a new set of categories as a new task.

Original: With a perfect task identiﬁer, they classify each task separately attest time.
Revised: With a perfect task identiﬁer, they classify each task separately at test time.

Original: The perfect identiﬁer is however often not available in practice [32].
Revised: The perfect identiﬁer is however often not available in practice [32].

Original: On the contrary,class-incremental methods [4, 8, 33, 36] attempt to recognize all categories observed so far at oncewithout the task identiﬁer, being more practical.
Revised: On the contrary, class-incremental methods [4, 8, 33, 36] attempt to recognize all categories observed so far at once without the task identiﬁer, being more practical.

Original: In the following, we describe class-incrementalmethods pertinent to ours.
Revised: In the following, we describe class-incremental methods pertinent to ours.

Original: Image classiﬁcation.
Revised: Image classiﬁcation.

Original: Incremental methods typically adopt a KD technique [15] in order to encourage a current model to imitate softmax probabilities [1, 33] or intermediate representations [8, 18]obtained from a previous one, retaining the discriminability for previous categories.
Revised: Incremental methods typically adopt a KD technique [15] in order to encourage a current model to imitate softmax probabilities [1, 33] or intermediate representations [8, 18] obtained from a previous one, retaining the discriminability for previous categories.

Original: Another lineof works focuses on preventing a classiﬁer from overﬁtting to new categories, by correcting biasedweights of a classiﬁer with a post-processing [36, 38] or normalizing a classiﬁer [18].
Revised: Another line of works focuses on preventing a classiﬁer from overﬁtting to new categories, by correcting biased weights of a classiﬁer with a post-processing [36, 39] or normalizing a classiﬁer [18].

Original: Similarly, our
Revised: Similarly, our

Original: ALI is beneﬁcial to alleviating the overﬁtting problem and maintaining the discriminative power forprevious categories, but differs in that it adaptively regularizes logit values during training.
Revised: ALI is beneﬁcial to alleviating the overﬁtting problem and maintaining the discriminative power for previous categories, but differs in that it adaptively regularizes logit values during training.

Original: In addition,we exploit latent features for replaying, while all the aforementioned methods rely on an image replaystrategy that entails practical issues such as data privacy and large memory requirements.
Revised: In addition, we exploit latent features for replaying, while all the aforementioned methods rely on an image replay strategy that entails practical issues such as data privacy and large memory requirements.

Original: To avoidthese problems, recent works employ a pseudo replay scheme that synthesizes images [20, 31, 34]or latent features [37] of previous categories.
Revised: To avoid these problems, recent works employ a pseudo replay scheme that synthesizes images [20, 31, 34] or latent features [38] of previous categories.

Original: However, they often require a generator [12], whichshould also be trained incrementally.
Revised: However, they often require a generator [12], which should also be trained incrementally.

Original: Instead of using raw images or synthesizing features, the workof [19] proposes to store latent features.
Revised: Instead of using raw images or synthesizing features, the work of [19] proposes to store latent features.

Original: It trains a feature adaptation network (FAN) mapping thelatent features into a new feature space.
Revised: It trains a feature adaptation network (FAN) mapping the latent features into a new feature space.

Original: The FAN might be sub-optimal, since the same networkapplies to all features without differentiating categories.
Revised: The FAN might be sub-optimal, since the same network applies to all features without differentiating categories.

Original: Adopting individual FANs for each categorymight address the problem, but this requires high computational cost and large memory footprint.
Revised: Adopting individual FANs for each category might address the problem, but this requires high computational cost and large memory footprint.

Original: Differently, we propose to train category-speciﬁc rotation matrices.
Revised: Differently, we propose to train category-speciﬁc rotation matrices.

Original: This is more efﬁcient and accurate,as a rotation matrix is light-weight and features of each category are updated separately.
Revised: This is more efﬁcient and accurate, as a rotation matrix is light-weight and features of each category are updated separately.

Original: Semantic segmentation.
Revised: Semantic segmentation.

Original: A relatively few methods address the task of ISS, which can further beclassiﬁed into regularization-based and replay-based approaches.
Revised: A relatively few methods address the task of ISS, which can further be classiﬁed into regularization-based and replay-based approaches.

Original: The ﬁrst approaches [2, 9, 30] focuson preserving knowledge for previous categories without using an experience replay.
Revised: The ﬁrst approaches [2, 9, 30] focus on preserving knowledge for previous categories without using an experience replay.

Original: For example,
Revised: For example,

Original: MiB [2] proposes to consider the semantic shift of a background category in ISS, and introduces
Revised: MiB [2] proposes to consider the semantic shift of a background category in ISS, and introduces

Original: CCE and CKD terms using probability calibration techniques.
Revised: CCE and CKD terms using probability calibration techniques.

Original: While the calibration techniques areintuitive and effective, there are still lack of theoretical explanations.
Revised: While the calibration techniques are intuitive and effective, there are still lack of theoretical explanations.

Original: We provide a gradient analysison CCE and CKD, and point out that CCE and CKD could disturb preserving knowledge for previouscategories and discriminating new categories from the background category, respectively.
Revised: We provide a gradient analysis on CCE and CKD, and point out that CCE and CKD could disturb preserving knowledge for previous categories and discriminating new categories from the background category, respectively.

Original: SDR [30]and PLOP [9] propose to further regularize a current model in a latent feature space.
Revised: SDR [30] and PLOP [9] propose to further regularize a current model in a latent feature space.

Original: Speciﬁcally,
Revised: Speciﬁcally,

Original: SDR minimizes distances between features of the same category [6], while PLOP distills intermediatefeatures using a multi-scale strip pooling technique [8, 17].
Revised: SDR minimizes distances between features of the same category [6], while PLOP distills intermediate features using a multi-scale strip pooling technique [8, 17].

Original: The second approaches [3, 26] rely onan image replay strategy.
Revised: The second approaches [3, 26] rely on an image replay strategy.

Original: RECALL [26] exploits web-crawled images with pseudo labels at eachincremental stage, while SSUL [3] maintains a small set of images for previous categories, togetherwith corresponding ground-truth masks.
Revised: RECALL [26] exploits web-crawled images with pseudo labels at each incremental stage, while SSUL [3] maintains a small set of images for previous categories, together with corresponding ground-truth masks.

Original: Both methods show state-of-the-art results at the cost oflarge memory footprint.
Revised: Both methods show state-of-the-art results at the cost of large memory footprint.

Original: Our approach differs in two aspects: (1) We exploit latent features forreplaying, which reduces the size of required memory signiﬁcantly; (2) Our method updates a featureextractor continually, and thus it is more ﬂexible than RECALL and SSUL freezing the extractor.
Revised: Our approach differs in two aspects: (1) We exploit latent features for replaying, which reduces the size of required memory signiﬁcantly; (2) Our method updates a feature extractor continually, and thus it is more ﬂexible than RECALL and SSUL freezing the extractor.

Original:  Method
Revised:  Method

Original: In this section, we ﬁrst describe the ISS task brieﬂy (Sec. 3.1), and then present our approachconsisting of two steps at every incremental stage.
Revised: In this section, we ﬁrst describe the ISS task brieﬂy (Sec. 3.1), and then present our approach consisting of three steps at every incremental stage.

Original: Speciﬁcally, we introduce ALI based on anin-depth analysis for CCE and CKD  (Sec. 3.2), and present a feature replay scheme using categoryspeciﬁc rotation matrices (Sec.3.3).
Revised: Speciﬁcally, we introduce ALI based on an in-depth analysis for CCE and CKD to train a ISS model (Sec. 3.2) and present a feature replay scheme using category-speciﬁc rotation matrices (Sec.3.3) to ﬁne-tune a classiﬁer using memorized features (Sec. 3.4).

Revised: Note that the last two steps are optional.

Original: Please refer to the supplement for  more analysis and detailedderivations.
Revised: Please refer to the supplementary material for detailed derivations, more analysis, and a pseudo code of our approach.

Original: 3.1 Problem statement
Revised: 3.1 Problem statement

Original: Following the common practice [2, 3, 9, 26, 30], we consider a series of T learning stages.
Revised: Following the common practice [2, 3, 9, 26, 30], we consider a series of T learning stages.

Original: Eachstage t has its own dataset, D t = { ( x i , y i ) } n t i =1 of size n t , where x and y are an image and acorresponding ground-truth mask, respectively.
Revised: Each stage t has its own dataset, D t = { ( x i , y i ) } n t i =1 of size n t , where x and y are an image and a corresponding ground-truth mask, respectively.

Original: For incremental stages ( i.e. , t > 1 ), we have twodisjoint sets, C t prev and C t new , for previously learned and novel categories, respectively.
Revised: For incremental stages ( i.e. , t > 1 ), we have two disjoint sets, C t prev and C t new , for previously learned and novel categories, respectively.

Original: Note that ground-truth masks are labeled only for the categories of C t new .
Revised: Note that ground-truth masks are labeled only for the categories of C t new .

Original: Formally, we denote by y ( p ) aground-truth label at position p , and deﬁne labeled regions as follows:
Revised: Formally, we denote by y ( p ) a ground-truth label at position p , and deﬁne labeled regions as follows:

Original: [Equation]
Revised: [Equation]

Original: where R c is a set of locations labeled as a category c , that is, R c
Revised: where R c is a set of locations labeled as a category c , that is, R c

Original: = { p | y ( p ) =
Revised: = { p | y ( p ) =

Original: c } .
Revised: c } .

Original: Our goal is to train a model that recognizes all categories observed so far, C t all , the union of thedisjoint sets, i.e. , C t all =
Revised: Our goal is to train a model that recognizes all categories observed so far, C t all , the union of the disjoint sets, i.e. , C t all =

Original: C t prev ∪ C t new .
Revised: C t prev ∪ C t new .

Original: In detail, our model consists of a feature extractor φ anda classiﬁer w .
Revised: In detail, our model consists of a feature extractor φ and a classiﬁer w .

Original: The feature extractor takes an image and outputs a convolutional feature map forprediction, φ t : x (cid:55)→ f t , where we denote by f t ( p ) ∈
Revised: The feature extractor takes an image and outputs a convolutional feature map for prediction, φ t : x (cid:55)→ f t , where we denote by f t ( p ) ∈

Original: R D a D -dimensional feature vector atposition p .
Revised: R D a D -dimensional feature vector at position p .

Original: A logit value for a category c is then computed by the dot product between a feature anda classiﬁer weight for the category c , w tc ∈ R D , as follows:
Revised: A logit value for a category c is then computed by the dot product between a feature and a classiﬁer weight for the category c , w tc ∈ R D , as follows:

Original: [Equation]
Revised: [Equation]

Original: where we omit bias terms of the classiﬁer for brevity.
Revised: where we omit bias terms of the classiﬁer for brevity.

Original: 3.2 Step1
Revised: 3.2 Step

Original: In the ﬁrst step, we initialize network weights of a model at the stage t with those for the previousstage t − 1 , and train a feature extractor and a classiﬁer of the current model with a correspondingdataset D t .
Revised: In the ﬁrst step, we initialize network weights of a model at the stage t with those for the previous stage t − 1 , and train a feature extractor and a classiﬁer of the current model with a corresponding dataset D t .

Original: A simple way to mitigate catastrophic forgetting for ISS is employing cross-entropy (CE)and KD [15] terms.
Revised: A simple way to mitigate catastrophic forgetting for ISS is employing cross-entropy (CE) and KD [15] terms.

Original: To this end, we compute probabilities for a category c at position p as follows:
Revised: To this end, we compute probabilities for a category c at position p as follows:

Original: [Equation]
Revised: [Equation]

Original: which are obtained by applying the softmax function to logit values across C t all and C t prev , respectively.
Revised: which are obtained by applying the softmax function to logit values across C t all and C t prev , respectively.

Original: The CE and KD terms for ISS are then deﬁned as follows:
Revised: The CE and KD terms for ISS are then deﬁned as follows:

Original: [Equation]
Revised: [Equation]

Original: where c ∗ =
Revised: where c ∗ =

Original: y ( p ) .
Revised: y ( p ) .

Original: The CE term is deﬁned for the labeled regions R  new only, indicating thatlogit values of previous categories always decrease.
Revised: The CE term is deﬁned for the labeled regions R t new only, indicating that logit values of previous categories always decrease.

Original: Exploiting the CE term alone is thus prone tooverﬁtting to new categories and leads to catastrophic forgetting.
Revised: Exploiting the CE term alone is thus prone to overﬁtting to new categories and leads to catastrophic forgetting.

Original: The KD term addresses this problemby transferring the discriminative power of a previous model, trained to classify previous categories,into the current one.
Revised: The KD term addresses this problem by transferring the discriminative power of a previous model, trained to classify previous categories, into the current one.

Original: Note that C t prev =
Revised: Note that C t prev =

Original: C t − 1 all , and the KD term computes softmax probabilitiesacross previous categories without considering new ones.
Revised: C t − 1 all , and the KD term computes softmax probabilities across previous categories without considering new ones.

Original: Instead of exploiting CE and KD terms, theseminal work [2] proposes to use CCE and CKD, widely adopted in ISS [2, 9, 30].
Revised: Instead of exploiting CE and KD terms, the seminal work [2] proposes to use CCE and CKD, widely adopted in ISS [2, 9, 30].

Original: In the following,we analyze gradients of CCE and CKD to better understand the inﬂuences on ISS, and introduce ALIto train our model.[Table]
Revised: In the following, we analyze gradients of CCE and CKD to better understand the inﬂuences on ISS, and introduce ALI to train our model.

Original: [Equation] [Table] [Equation] [Table]
Revised: Conditions

Original: [Equation] [Table] [Equation] [Table]
Revised: Gradients

Original: [Equation]
Revised: [Equation]

Original: CCE.
Revised: CCE.

Original: The CCE loss [2] also computes a probability for unlabeled regions by summing the probabilities over all previouscategories as follows:
Revised: The CCE loss [2] additionally computes a probability for unlabeled regions by summing the probabilities over all previous categories as follows:

Original: [Equation]
Revised: [Equation]

Original: where p t cce ( p ) = (cid:80)
Revised: where p t cce ( p ) = (cid:80)

Original: k ∈ C t prev p tk ( p ) .
Revised: k ∈ C t prev p tk ( p ) .

Original: This marginal differencefrom the vanilla CE term brings a signiﬁcant improvement on ISS.
Revised: This marginal difference from the vanilla CE term brings a signiﬁcant improvement on ISS.

Original: To analyze the reason behind theimprovement, we summarize in Table 1 gradients of L CCE w.r.t z tc .
Revised: To analyze the reason behind the improvement, we summarize in Table 1 gradients of L CCE w.r.t z tc .

Original: The ﬁrst two rows show that CCEupdates logit values for the labeled regions in the same way as in L CE .
Revised: The ﬁrst two rows show that CCE updates logit values for the labeled regions in the same way as in L CE .

Original: We can see from last tworows that CCE also provides gradients for the unlabeled regions.
Revised: We can see from last two rows that CCE also provides gradients for the unlabeled regions.

Original: Speciﬁcally, the third row showsthat CCE reduces a logit value for new categories by the corresponding probability p tc , if p / ∈ R t new .
Revised: Speciﬁcally, the third row shows that CCE reduces a logit value for new categories by the corresponding probability p tc , if p / ∈ R t new .

Original: This is reasonable, since all features on those regions do not at least belong to new categories.
Revised: This is reasonable, since all features on those regions do not at least belong to new categories.

Original: CCEthus helps to avoid the overﬁtting problem, alleviating catastrophic forgetting.
Revised: CCE thus helps to avoid the overﬁtting problem, alleviating catastrophic forgetting.

Original: Considering the factthat q tc is always larger than p tc (See the supplement), we can see from the last row that CCE raises logit values for all previous categories by q tc − p tc in the unlabeled regions.
Revised: Considering the fact that q tc is always larger than p tc (See the supplementary material), we can see from the last row that CCE raises logit values for all previous categories by q tc − p tc in the unlabeled regions.

Original: This prevents the logitvalues of previous categories from continuing to decrease in the unlabeled regions, which is howevereffective only when predictions of the current model are correct.
Revised: This prevents the logit values of previous categories from continuing to decrease in the unlabeled regions, which is however effective only when predictions of the current model are correct.

Original: Otherwise, when predictions areincorrect, CCE rather raises logit values of the wrongly predicted categories, aggravating the incorrectpredictions.
Revised: Otherwise, when predictions are incorrect, CCE rather raises logit values of the wrongly predicted categories, aggravating the incorrect predictions.

Original: CKD.
Revised: CKD.

Original: Assuming that categories of C t new are likely to be labeled as a background one in the previousstage, the CKD loss [2] is deﬁned as follows:
Revised: Assuming that categories of C t new are likely to be labeled as a background one in the previous stage, the CKD loss [2] is deﬁned as follows:

Original: [Equation]
Revised: [Equation]

Original: where p t ckd ( p ) =
Revised: where p t ckd ( p ) =

Original: (cid:80) k ∈{ bg }∪ C t new p
Revised: (cid:80) k ∈{ bg }∪ C t new p

Original: tk ( p ) .
Revised: tk ( p ) .

Original: For previous categories except the background, this termenables transferring knowledge from p t − 1 k to p tk directly.
Revised: For previous categories except the background, this term enables transferring knowledge from p t − 1 k to p tk directly.

Original: Similarly, a vanilla KD term in Eq.
Revised: Similarly, a vanilla KD term in Eq.

Original: (4) alsoencourages a current model to output probabilities similar to p t − 1 k for k ∈ C t prev , but it distills theknowledge from p t − 1 k into q tk without considering logit values for new categories.
Revised: (4) also encourages a current model to output probabilities similar to p t − 1 k for k ∈ C t prev , but it distills the knowledge from p t − 1 k into q tk without considering logit values for new categories.

Original: To compare KDand CKD, we compute gradients w.r.t z tc for c ∈ C t prev \{ bg } as follows:
Revised: To compare KD and CKD, we compute gradients w.r.t z t c for c ∈ C t prev \{ bg } as follows:

Original: [Equation]
Revised: [Equation]

Original: Here we provide two cases to explain the reason why CKD better transfers the knowledge for previouscategories except the background: (1) When q tc and p t − 1 c are equal, the gradient of KD w.r.t z tc is zero,indicating that z tc remains the same.
Revised: Here we provide two cases to explain the reason why CKD better transfers the knowledge for previous categories except the background: (1) When q tc and p t − 1 c are equal, the gradient of KD w.r.t z tc is zero, indicating that z tc remains the same.

Original: However, note that p tc is lower than p t − 1 c in this case, since p tc isalways lower than q tc .
Revised: However, note that p tc is lower than p t − 1 c in this case, since p tc is always lower than q tc .

Original: This suggests that z tc should rather increase in order that p tc follows the targetprobability p t − 1 c .
Revised: This suggests that z tc should rather increase in order that p tc follows the target probability p t − 1 c .

Original: The gradient of CKD w.r.t z
Revised: The gradient of CKD w.r.t z

Original: tc has a negative value in the same case ( i.e. , q tc = p t − 1 c ),suggesting that z tc increases by gradient descent.
Revised: tc has a negative value in the same case ( i.e. , q tc = p t − 1 c ), suggesting that z tc increases by gradient descent.

Original: (2) When the current model already provides thesame probability as the target one ( i.e. , p tc = p t − 1 c ), it is unnecessary to adjust z tc .
Revised: (2) When the current model already provides the same probability as the target one ( i.e. , p tc = p t − 1 c ), it is unnecessary to adjust z tc .

Original: The gradientof KD w.r.t z
Revised: The gradient of KD w.r.t z

Original: tc has a positive value, as q tc is larger than p tc (or equivalently p t − 1 c in this case).
Revised: tc has a positive value, as q tc is larger than p tc (or equivalently p t − 1 c in this case).

Original: Thisindicates that z tc decreases by gradient descent, and p tc in turn becomes lower than p t − 1 c .
Revised: This indicates that z tc decreases by gradient descent, and p tc in turn becomes lower than p t − 1 c .

Original: On the otherhand, CKD maintains z tc the same, since its gradient w.r.t z tc is zero.
Revised: On the other hand, CKD maintains z tc the same, since its gradient w.r.t z tc is zero.

Original: These examples describe theeffectiveness of CKD.
Revised: These examples describe the effectiveness of CKD.

Original: On the contrary, CKD also has negative effects on ISS.
Revised: On the contrary, CKD also has negative effects on ISS.

Original: Let us suppose thegradients of CKD w.r.t z tc for c ∈ { bg } ∪ C t new as follows:
Revised: Let us suppose the gradients of CKD w.r.t z tc for c ∈ { bg } ∪ C t new as follows:

Original: [Equation]
Revised: [Equation]

Original: The gradient of CKD  in Eq.
Revised: The problem of CKD lies in how it transfers the knowledge from p t − 1 bg into p tbg . In particular, CKD makes p t ckd to imitate p t − 1 bg , instead of directly distilling from p t − 1 bg to p tbg .

Original: (8) has a negative value, when p t ckd is lower than p t − 1 bg .
Revised: For example, when p t ckd is lower than p t − 1 bg , the gradient of CKD in Eq. (8) has a negative value.

Original: Thus, logit valuesfor background and new categories increase by gradient descent.
Revised: Thus, logit values for background and new categories increase by gradient descent.

Original: This in turn raises probabilities forbackground and new categories all together, disturbing discriminating the new categories from thebackground at training time.
Revised: This in turn raises probabilities for background and new categories all together, disturbing discriminating the new categories from the background at training time.

Original: In case of p t ckd > p t − 1 bg , CKD reduces the logit values for backgroundand new categories, which is however problematic when p tbg < p t − 1 bg .
Revised: In case of p t ckd > p t − 1 bg , CKD reduces the logit values for background and new categories, which is however problematic when p tbg < p t − 1 bg .

Original: In this case, the logit value forthe background category z t bg should increase.
Revised: Note that the logit value for the background category z tbg should rather increase in this case.

Original: [Table]
Revised: [Table]

Original: ALI.
Revised: ALI.

Original: We have shown that CCE alleviates catastrophic forgetting, while CKD better guides transferring the knowledgeof a previous model than KD.
Revised: We have shown that CCE alleviates catastrophic forgetting, while CKD better guides transferring the knowledge of a previous model than KD.

Original: To avoid the negative effects of
Revised: To avoid the negative effects of

Original: CCE and CKD, we present in Table 2 a new form of gradientsw.r.t z tc .
Revised: CCE and CKD, we present in Table 2 a new form of gradients w.r.t z tc .

Original: The ﬁrst row is identical to the third one of Table 1that alleviates catastrophic forgetting.
Revised: The ﬁrst row is identical to the third one of Tablethat alleviates catastrophic forgetting.

Original: The second row is similar to the gradients of CKD in Eq.
Revised: The second row is similar to the gradients of CKD in Eq.

Original: (7)that better capture knowledge for previous categories.
Revised: (7) that better capture knowledge for previous categories.

Original: Note that CKD excludes the backgroundcategory in Eq.
Revised: Note that CKD excludes the background category in Eq.

Original: (7), since it assumes that new categories of the current stage ( C t new ) are marked as thebackground one at the previous stage.
Revised: (7), since it assumes that new categories of the current stage ( C t new ) are marked as the background one at the previous stage.

Original: Differently, ours enables computing gradients for all previouscategories (See the second row), since it does not require the assumption.
Revised: Differently, ours enables computing gradients for all previous categories including the background one (See the second row), since it does not require the assumption.

Original: Accordingly, this form ofgradients incorporates the advantages of CCE and CKD, while discarding the negative effects.
Revised: Accordingly, this form of gradients incorporates the advantages of CCE and CKD, while discarding the negative effects.

Original: Usingthe gradients in Table 2, we deﬁne ALI as follows:[Equation] [Table]
Revised: Integrating the gradients in Table 2 w.r.t z tc , we deﬁne ALI as follows:

Original: [Equation]

Original: [Table]

Original: [Equation]
Revised: [Equation]

Original: The ﬁrst term applies the log-sum-exp function to logit values across C t all , approximating themaximum logit value over C t all .
Revised: The ﬁrst term applies the log-sum-exp function to logit values across C t all , approximating the maximum logit value over C t all .

Original: The second term computes a weighted average of logit valuesfor C t prev with probabilities of a previous model.
Revised: The second term computes a weighted average of logit values for C t prev with probabilities of a previous model.

Original: In this context, ALI can be viewed as minimizingthe difference between the maximum logit value and the weighted average adaptively.
Revised: In this context, ALI can be viewed as minimizing the difference between the maximum logit value and the weighted average adaptively.

Original: That is, itreduces the maximum logit value, while raising the logit values for previous categories.
Revised: That is, it reduces the maximum logit value, while raising the logit values for previous categories.

Original: Note that
Revised: Note that

Original: ALI does not require computing either p t cce in CCE or p t ckd in CKD [2].
Revised: ALI does not require computing either p t cce in CCE or p t ckd in CKD [2].

Original: Training.
Revised: Training.

Original: To train our model, we use CE and ALI terms for labeled and unlabeled regions, respectively.
Revised: To train our model, we use CE and ALI terms for labeled and unlabeled regions, respectively.

Original: We also apply a vanilla KD term for labeled regions to further regularize our model.
Revised: We also apply a vanilla KD term for labeled regions to further regularize our model.

Original: Anoverall objective for the ﬁrst step is deﬁned as follows:
Revised: An overall objective for the ﬁrst step is deﬁned as follows:

Original: [Equation]
Revised: [Equation]

Original: where λ ALI and λ KD are balance parameters.
Revised: where λ ALI and λ KD are balance parameters.

Original: We denote by 1 [ · ] an indicator function whose output is1 if the argument is true, and 0 otherwise. 
Revised: We denote by 1 [ · ] an indicator function whose output is 1 if the argument is true, and 0 otherwise. 3.3 Step

Original: Memorizing features.
Revised: After training our model, we ﬁrst extract features of new categories in order to replay them in subsequent stages. Then, we compensate a distribution shift of memorized features, which are extracted in the previous stage t − 1 , before using them to ﬁne-tune a classiﬁer in the third step. To be speciﬁc, we exploit category-speciﬁc rotation matrices to update memorized features of each category separately. In the following, we provide detailed descriptions for memorizing features and training matrices.

Revised: Note that we freeze both a feature extractor φ t and a classiﬁer w t for the second step.

Original: After training our model, we extract features for prediction  and store themto replay for the next stage.
Revised: Memorizing features. We extract features for prediction ( i.e. , f t ) and store them to replay in subsequent stages.

Original: Speciﬁcally, given input images containing one of new categories, afeature extractor ﬁrst produces feature maps.
Revised: Speciﬁcally, given input images containing one of new categories, a feature extractor ﬁrst produces feature maps.

Original: For each image, features for each category are thenaveraged using a ground-truth mask.
Revised: For each image, features for each category are then averaged using a ground-truth mask.

Original: This process is repeated until the number of features for eachcategory reaches a preset number S .
Revised: This process is repeated until the number of features for each category reaches a preset number S .

Original: That is, we memorize S features for each category.
Revised: That is, we memorize S features for each category.

Original: See thesupplement for detailed descriptions.

Original: 3.3 Step 2

Original: After the ﬁrst step, we freeze a feature extractor φ t and ﬁne-tune a classiﬁer w t with a currentdataset D t along with memorized features extracted from the previous stage t − 1 . The memorizedfeatures, however, do not lie in the current feature space. To address this, we exploit category-speciﬁcrotation matrices to update features of each category separately, and present two loss terms to trainthe matrices. In the following, we provide detailed descriptions for training matrices and ﬁne-tuningclassiﬁers.
Revised: We provide the pseudo code in the supplementary material.

Original: Training matrices.
Revised: Training matrices.

Original: Memorized features, extracted in the previous stage t − 1 , are not compatiblewith a current classiﬁer w t .
Revised: Memorized features, extracted in the previous stage t − 1 , are not compatible with a current classiﬁer w t .

Original: To address this, the work of [19] in image classiﬁcation proposes toexploit two-layer perceptrons, called FAN, where the number of parameters is roughly 32 D 2 .
Revised: To address this, the work of [19] in image classiﬁcation proposes to exploit two-layer perceptrons, called FAN, where the number of parameters is roughly 32 D 2 .

Original: Twomain limitations of FAN are as follows: (1) It updates all features without considering categories.
Revised: Two main limitations of FAN are as follows: (1) It updates all features without considering categories.

Original: Adopting separate FANs for each category alleviates this issue, but it is computationally expensive.
Revised: Adopting separate FANs for each category alleviates this issue, but it is computationally expensive.

Original: FAN ignores the relations between memorized features.
Revised: FAN ignores the relations between memorized features.

Original: The structural information in the featurespace is crucial for the generalization ability of classiﬁers.
Revised: The structural information in the feature space is crucial for the generalization ability of classiﬁers.

Original: To address these problems, we propose totrain category-speciﬁc rotation matrices to update features of each category separately.
Revised: To address these problems, we propose to train category-speciﬁc rotation matrices to update features of each category separately.

Original: The rotationtransform enables maintaining the relations between features within the same category, while thenumber of parameters for each matrix is 0 .
Revised: The rotation transform enables maintaining the relations between features within the same category, while the number of parameters for each matrix is 0 .

Original: 5( D 2 − D ) only (See the supplement).
Revised: 5( D 2 − D ) only (See the supplementary material).

Original: To this end, weﬁrst deﬁne a skew-symmetric matrix S c of size D × D for the category c ∈ C t prev .
Revised: To this end, we ﬁrst deﬁne a skew-symmetric matrix S c of size D × D for the category c ∈ C t prev .

Original: A rotation matrixfor the category c is then deﬁned using the Cayley transform as follows: 
Revised: A rotation matrix for the category c is then deﬁned using the Cayley transform as follows: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where I indicates an identity matrix.
Revised: where I indicates an identity matrix.

Original: Learning the parameters for the matrices is challenging, sincetraining samples of D t are labeled only for C t new .
Revised: Learning the parameters for the matrices is challenging, since training samples of D t are labeled only for C t new .

Original: To handle this, we extract features f t − 1 from aprevious feature extractor φ t − 1 , and compute a correlation score as follows:
Revised: To handle this, we extract features f t − 1 from a previous feature extractor φ t − 1 , and compute a correlation score as follows:

Original: [Equation]
Revised: [Equation]

Original: where we denote by m c ( s )  the s -th item in memorized features of the category c .
Revised: where we denote by m c ( s ) ∈ R D the s -th item in memorized features of the category c .

Original: This allows toidentify features associated with previous categories C t prev , since f t − 1 and m c ( s ) share the samefeature space.
Revised: This allows to identify features associated with previous categories C t prev , since f t − 1 and m c ( s ) share the same feature space.

Original: We then apply the softmax function over the correlation score as follows:
Revised: We then apply the softmax function over the correlation score as follows:

Original: [Equation]
Revised: [Equation]

Original: where τ is a temperature controlling the sharpness of σ c .
Revised: where τ is a temperature controlling the sharpness of σ c .

Original: Note that both features f t − 1 and f t atposition p encode the same semantic information.
Revised: Note that both features f t − 1 and f t at position p encode the same semantic information.

Original: Using this fact, we deﬁne prototypes of thecategory c for previous and current stages, r t − 1 c and r tc , respectively, by computing a weightedaverage of each feature map, as follows:
Revised: Using this fact, we deﬁne prototypes of the category c for previous and current stages, r t − 1 c and r tc , respectively, by computing a weighted average of each feature map, as follows:

Original: [Equation]
Revised: [Equation]

Original: We can match the prototypes for each category c via the corresponding rotation matrix R c .
Revised: We can match the prototypes for each category c via the corresponding rotation matrix R c .

Original: Namely,each matrix R c rotates a previous prototype r t − 1 c to align it with a current one r tc .
Revised: Namely, each matrix R c rotates a previous prototype r t − 1 c to align it with a current one r tc .

Original: To train thematrix R c , we deﬁne an objective function as follows:
Revised: To train the matrix R c , we deﬁne an objective function as follows:

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original:  where we denote by L FID and L REG ﬁdelity and regularization terms, respectively, balanced by theparameter λ FID .
Revised: (15) where we denote by L FID and L REG ﬁdelity and regularization terms, respectively, balanced by the parameter λ ROT .

Original: The ﬁdelity term maximizes cosine similarity as follows:
Revised: The ﬁdelity term maximizes cosine similarity as follows:

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: where ˆ r c
Revised: where ˆ r c

Original: = R c r t − 1 c .
Revised: = R c r t − 1 c .

Original: This encourages the matrices R c to align ˆ r c with r tc .
Revised: This encourages the matrices R c to align ˆ r c with r tc .

Original: The regularization termenforces the rotated prototypes ˆ r c to be compatible with the current classiﬁer w t to better guide thealignment process.
Revised: The regularization term enforces the rotated prototypes ˆ r c to be compatible with the current classiﬁer w t to better guide the alignment process.

Original: To this end, we compute a CE loss using the softmax classiﬁer w t as follows:
Revised: To this end, we compute a CE loss using the softmax classiﬁer w t as follows:

Original: [Equation]
Revised: [Equation]

Original: Note that we freeze both a feature extractor and a classiﬁer when training rotation matrices. Fine-tuning.
Revised: 3.4 Step

Original: After training rotation matrices, we  update memorized features of each category asfollows:
Revised: In the third step, we ﬁrst update memorized features of each category as follows:

Original: [Equation]
Revised: [Equation]

Original: The updated features along with training samples of D t are then used to ﬁne-tune a classiﬁer w t withthe following objective:
Revised: The updated features along with training samples of D t are then used to ﬁne-tune a classiﬁer w t with the following objective:

Original: [Equation]
Revised: [Equation]

Original: We denote by L FL and L MEM focal loss (FL) [24] and CE terms, respectively, deﬁned as follows:
Revised: We denote by L FL and L MEM focal loss (FL) [24] and CE terms, respectively, deﬁned as follows:

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: where we set α to 2 by default.
Revised: where we set α to 2 by default.

Original: Following [3, 9, 26, 30], we mark unlabeled regions in the trainingsamples of D t as predictions obtained from a previous model on-the-ﬂy.
Revised: Following [3, 9, 26, 30], we mark unlabeled regions in the training samples of D t as predictions obtained from a previous model on-the-ﬂy.

Revised: Note that we freeze a feature extractor and the rotation matrices for ﬁne-tuning the classiﬁer.

Original:  Experiments
Revised:  Experiments

Original: In this section, we present a quantitative comparison between our method and the state of the art, andshow ablation studies.
Revised: In this section, we present a quantitative comparison between our method and the state of the art, and show ablation studies.

Original: More results including qualitative comparisons can be found in the supplement.
Revised: More results including qualitative comparisons can be found in the supplementary material.

Revised: [Table]

Original: 4.1 Implementation details
Revised: 4.1 Implementation details

Original: Dataset and evaluation.
Revised: Dataset and evaluation.

Original: We evaluate our method on standard ISS benchmarks (PASCAL VOC [10]and ADE20K [39]).
Revised: We evaluate our method on standard ISS benchmarks (PASCAL VOC [10] and ADE20K [40]).

Original: PASCAL VOC provides 10 , 582 training [13] and 1 , 449 validation samples with20 object and one background categories, while ADE20K consists of 20 , 210 and 2 , 000 samples fortraining and validation, respectively, with 150 object/stuff categories.
Revised: PASCAL VOC provides 10 , 582 training [13] and 1 , 449 validation samples with 20 object and one background categories, while ADE20K consists of 20 , 210 and 2 , 000 samples for training and validation, respectively, with 150 object/stuff categories.

Original: There are three incremental[Table] scenarios for each dataset, where we denote by each scenario A - B ( C ).
Revised: There are three incremental scenarios for each dataset, where we denote by each scenario A - B ( C ).

Original: A, B and C indicate thenumber of categories at a base stage, the total number of novel categories, and the number ofincremental stages, respectively.
Revised: A, B and C indicate the number of categories at a base stage, the total number of novel categories, and the number of incremental stages, respectively.

Original: We follow the same scenarios as in [2, 3, 9, 26, 30], unless otherwisespeciﬁed.
Revised: We follow the same scenarios as in [2, 3, 9, 26, 30], unless otherwise speciﬁed.

Original: Following the common practice [3, 9], we focus on an overlapped setting, where unlabeledregions could contain either previous or future categories.
Revised: Following the common practice [3, 9], we focus on an overlapped setting, where unlabeled regions could contain either previous or future categories.

Original: For evaluation, we report IoU scoreson the validation set for each dataset, and do not exploit test-time augmentation or dense CRFtechniques [22].
Revised: For evaluation, we report IoU scores on the validation set for each dataset, and do not exploit test-time augmentation or dense CRF techniques [22].

Original: We denote by mIoU base and mIoU new  the mean IoU(mIoU) scores over base andnew categories, respectively.
Revised: We denote by mIoU base , mIoU new , and mIoU the mean IoU scores over base, new, and all categories, respectively.

Original: Computing an mIoU score over all categories  is typical, but we havefound that this does not reﬂect IoU scores for new categories well, and those for base categoriesmostly inﬂuence evaluation results.
Revised: Computing an IoU score over all categories ( i.e. mIoU) is typical, but we have found that this does not reﬂect IoU scores for new categories well.

Original: To address this,  we propose to use the harmonic mean (hIoU) ofmIoU base and mIoU new .
Revised: To address this, as in the evaluation protocol of zero-shot learning methods ( e.g. , [37]), we propose to use the harmonic mean (hIoU) of mIoU base and mIoU new .

Original: For all experiments, we report scores averaged over 3 runs.
Revised: For all experiments, we report scores averaged overruns ( i.e. , different random seeds).

Original: Training.
Revised: Training.

Original: Following [2, 3, 9, 26, 30], we adopt DeepLab-V3 [5] with ResNet-101 [14].
Revised: Following [2, 3, 9, 26, 30], we adopt DeepLab-V3 [5] with ResNet-101 [14].

Original: ResNet-101is initialized with pre-trained weights for classiﬁcation on ImageNet [7].
Revised: ResNetis initialized with pre-trained weights for classiﬁcation on ImageNet [7].

Original: We use the SGD optimizerwith an initial learning rate set to 1e-2 and 1e-3 for base and incremental stages, respectively.
Revised: We use the SGD optimizer with an initial learning rate set to 1e-2 and 1e-3 for base and incremental stages, respectively.

Original: DeepLab-V3 is trained for 30 and 60 epochs at a base stage ( t = 1 ) on PASCAL VOC [10] and
Revised: DeepLab-V3 is trained for 30 and 60 epochs at a base stage ( t = 1 ) on PASCAL VOC [10] and

Original: ADE20K [39], respectively.
Revised: ADE20K [40], respectively.

Original: For each incremental stage ( t > 1 ), we perform a cross-validation tochoose the number of epochs on PASCAL VOC, while ﬁxing it to 60 on ADE20K.
Revised: For each incremental stage ( t > 1 ), we perform a cross-validation to choose the number of epochs on PASCAL VOC, while ﬁxing it to 60 on ADE20K.

Original: We train rotationmatrices for 10 epochs using the SGD optimizer with an initial learning rate of 1e-2, and ﬁx a presetnumber S and a temperature value τ to 1 , 000 and 10 for all experiments.
Revised: We train rotation matrices for 10 epochs using the Adam optimizer with an initial learning rate of 1e-3, and ﬁx a preset number S and a temperature value τ to 1 , 000 and 10 for all experiments.

Original: We ﬁne-tune a classiﬁerfor 1 epoch using the SGD optimizer with an initial learning rate of 1e-3.
Revised: We ﬁne-tune a classiﬁer for 1 epoch using the SGD optimizer with an initial learning rate of 1e-3.

Original: For all experiments, weadjust the learning rate by the poly schedule.
Revised: For all experiments, we adjust the learning rate by the poly schedule.

Original: We provide a detailed description of hyperparametersettings in the supplement.
Revised: We provide a detailed description of hyperparameter settings in the supplementary material.

Original: 4.2 Comparison with the state of the art
Revised: 4.2 Comparison with the state of the art

Original: ADE20K.
Revised: ADE20K.

Original: We compare in Table 3 our approach with state-of-the-art methods, including MiB [2],
Revised: We compare in Table 3 our approach with state-of-the-art methods, including MiB [2],

Original: PLOP [9], and SSUL [3].
Revised: PLOP [9], and SSUL [3].

Original: Note that RECALL [26] is not designed to handle stuff categories, andresults on ADE20K [39] are not available.
Revised: Note that RECALL [26] is not designed to handle stuff categories, and results on ADE20K [40] are not available.

Original: From this table, we have three ﬁndings as follows: (1) Ourapproach exploiting the ﬁrst step only, denoted by Ours-S1 , already outperforms all other methodsin terms of both mIoU and hIoU scores by signiﬁcant margins for all scenarios.
Revised: From this table, we have three ﬁndings as follows: (1) Our approach exploiting the ﬁrst step only, denoted by ALIFE, already outperforms all other methods in terms of both mIoU and hIoU scores by signiﬁcant margins for all scenarios.

Original: This validates theeffectiveness of our approach without memorizing features.
Revised: This validates the effectiveness of our approach without memorizing features.

Original: In particular, we can see that OursS1 even  outperforms SSUL-M [3] that memorizes 300 images along with ground-truth labels forreplaying.
Revised: In particular, we can see that ALIFE even [Table] outperforms SSUL-M [3] that memorizes 300 images along with ground-truth labels for replaying.

Original: A plausible reason is that SSUL freezes a feature extractor, limiting the ﬂexibility to dealwith new categories.
Revised: A plausible reason is that SSUL freezes a feature extractor, limiting the ﬂexibility to deal with new categories.

Original: (2) Ours-S1 shows substantial IoU gains over MiB [2] using CCE and CKDfor all scenarios.
Revised: (2) ALIFE shows substantial IoU gains over MiB [2] using CCE and CKD for all scenarios.

Original: This veriﬁes that both CCE and CKD are not always helpful for ISS.
Revised: This veriﬁes that both CCE and CKD are not always helpful for ISS.

Original: ALI is freefrom the limitations of CCE and CKD, and it allows our model to better learn new categories withoutforgetting previous ones.
Revised: ALI is free from the limitations of CCE and CKD, and it allows our model to better learn new categories without forgetting previous ones.

Original: (3) Our approach memorizing features, denoted by Ours-S2 , improves theperformance over Ours-S1 in terms of all metrics for all scenarios.
Revised: (3) Our approach memorizing features, denoted by ALIFE-M, improves the performance over ALIFE in terms of all metrics for all scenarios.

Original: Note that SSUL-M even performsworse than SSULwithout replaying for 100-50(1) and 100-50(6) cases.
Revised: Note that SSUL-M even performs worse than SSUL for 100-50(1) and 100-50(6) cases.

Original: Considering that we rely onat least 58 times less memory requirements than SSUL-M on ADE20K, the gains from memorizingfeatures are remarkable compared to those of SSUL-M over SSUL.[Table]
Revised: Considering that we rely on at least 9 times less memory requirements than SSUL-M for 100-50(1) and 50-100(2) cases, the gains from memorizing features are remarkable compared to those of SSUL-M over SSUL.

Original: PASCAL VOC.
Revised: PASCAL VOC.

Original: We show in Table 4 quantitative results on PASCAL VOC [10].
Revised: We show in Table 4 quantitative results on PASCAL VOC [10].

Original: Note that acomparison of SSUL [3] (SSUL-M) and other methods including ours is not fair, since it additionallyexploits an off-the-shelf saliency detector [16] that brings signiﬁcant improvements on PASCAL
Revised: Note that a comparison of SSUL [3] (SSUL-M) and other methods including ours is not fair, since it additionally exploits an off-the-shelf saliency detector [16] that brings signiﬁcant improvements on PASCAL

Original: VOC.
Revised: VOC.

Original: From this table, we can see that Ours-S1 outperforms  other methods in terms of hIoU scores forall scenarios, further demonstrating the effectiveness of our approach without memorizing features.
Revised: From this table, we can see that ALIFE outperforms all other methods in terms of hIoU scores for 20-1(1) and 16-5(1) scenarios, further demonstrating the effectiveness of our approach without memorizing features.

Original: We can also see that Ours-S2 gives substantial gains over Ours-S1 in terms of all metrics for allscenarios.
Revised: We can also see that ALIFE-M gives substantial gains over ALIFE in terms of all metrics for all scenarios.

Original: In particular, the largest mIoU and hIoU gains of 3.85% and 6.61%, respectively, arereported on 16-5(5).
Revised: In particular, the largest mIoU and hIoU gains of 2.18% and 3.60%, respectively, are reported on 16-5(5).

Original: This conﬁrms once again that our feature replay scheme is effective even for themost challenging scenario on PASCAL VOC.
Revised: This conﬁrms once again that our feature replay scheme is effective even for the most challenging scenario on PASCAL VOC.

Original: SSUL-M [3] and RECALL [26] largely outperformours only for 16-5(5), but they require at least 604 and 19 times more memory footprint than Ours-S2 ,respectively.
Revised: SSUL-M [3] and RECALL [26] largely outperform ours only for 16-5(5), but they require at least 604 and 15 times more memory footprint than ALIFE-M, respectively.

Original: Moreover, RECALL has difﬁculty for handling stuff categories, and SSUL performspoorly on ADE20K [39], where the saliency detector is not applicable.
Revised: Moreover, RECALL has difﬁculty for handling stuff categories, and SSUL performs poorly on ADE20K [40], where the saliency detector is not applicable.

Original: Our approach is versatile inthat it is free to handle stuff categories without performance degradation on ADE20K
Revised: Our approach is versatile in that it is free to handle stuff categories without performance degradation on ADE20K

Original: (See Table 3).
Revised: (See Table 3).

Original: 4.3 Ablation study
Revised: 4.3 Discussion

Original: [Table]
Revised: [Figure]

Revised: The ordering of categories.

Revised: To show the robustness of our approach to different category orderings, we generate two random category orderings for 165(1) and 20-1(1) cases, and report the mean and standard deviation of hIoU scores over three orderings: one alphabetical and two random orderings.

Revised: We compare in Fig.

Revised: 2 our approach without memorizing fea- tures, denoted by ALIFE, to MiB [2] and PLOP [9].

Revised: Results of MiB and PLOP are obtained by running the source codes provided by the authors.

Revised: We can clearly see that ALIFE produces better results than other methods on both 16-5(1) and 20-1(1) cases.

Revised: [Table]

Original:  We show in Table 5(a) an ablation analysis on different lossterms of our approach without memorizing features (Eq.
Revised: Ablation study on the ﬁrst step. We show in Table 5 an ablation analysis on different loss terms of our approach without memorizing features (Eq.

Original: From this table, we can see that using the CE term alone performs poorly due to catastrophic forgetting.
Revised: The ﬁrst row shows that using the CE term alone performs poorly due to catastrophic forgetting.

Original: We can also seethat ALI alleviates the catastrophic forgetting problem remarkably in the second row.
Revised: We can see from the second row that ALI alleviates the catastrophic forgetting problem remarkably.

Original: The last row shows that  applying the 
Revised: The third row shows that additionally using the KD term for labeled regions on top of CE and ALI terms further gives IoU gains. On the other hand, we can see from the fourth row that applying the KD term for unlabeled regions rather degrades the performance.

Original:  KD term for labeled regions further gives IoU gains.
Revised: Since our ALI already encourages a current model to imitate knowledge of a previous one for unlabeled regions, we conjecture that additionally applying the KD term for those regions is redundant. The last row shows that using the KD term for all regions is beneﬁcial to improving the performance slightly. [Table] Ablation study on the third step.

Original: We reportin Table 5(b) IoU scores using different losses  in the secondstep (Eq.
Revised: We report in Table 6 IoU scores for different losses of our method in the third step (Eq.

Original: The ﬁrst row shows that the FL term alonedegrades the performance.
Revised: From the ﬁrst four rows, we can see that 1) using CE or FL terms alone degrade the performance.

Original: This is because training samples ofa current dataset mainly contain new categories, causing a classimbalance between previous and new categories.
Revised: This is because training samples of a current dataset mainly contain new categories, causing the class imbalance between previous and new categories; 2) FL provides better results than CE; 3)

Revised: The pseudo labeling strategy works favorably for both CE and FL, mitigating the imbalance.

Original: The secondrow shows that  replaying 1K features for each previous category lessens the inﬂuence of the classimbalance problem.
Revised: From the last three rows, we can see that 1) replaying 1K features for each previous category lessens the inﬂuence of the class imbalance problem signiﬁcantly and 2) ALI is also helpful to alleviate the imbalance problem, further boosting the performance. Limitation. Our approach to memorizing features for an experience replay reduces memory requirements signiﬁcantly and avoids data privacy issues. Nonetheless, it requires more memory than other methods [2, 9, 30] that do not adopt the experience replay.

Original: From the last two rows, we can see that ALI  is also helpful to alleviate theimbalance problem and it further boosts the performance along with replaying features.
Revised: Since our approach to using ALI without memorizing features shows state-of-the-art results on standard ISS benchmarks [10, 40], we believe that ALI could give useful insights for developing ISS methods that do not rely on the replay strategy. It is also worth noting that all existing methods including ours assume that newly incoming samples are clean and reliable. However, in practice, training samples of a new task might be biased and unreliable. This raises new concerns: 1) addressing noisy samples during training and 2) memorizing clean samples only.

Revised: Handling these potential risks would also be an interesting future direction for ISS.

Original:  Conclusion
Revised:  Conclusion

Original: We have introduced a new ISS method, ALIFE, that alleviates catastrophic forgetting and reducesmemory requirements for an experience replay.
Revised: We have introduced a new ISS method, ALIFE, that alleviates catastrophic forgetting and reduces memory requirements for an experience replay.

Original: First, we have presented a gradient analysis of CCEand CKD for better understanding the effects on catastrophic forgetting, and have proposed ALIincorporating the merits of CCE and CKD.
Revised: First, we have presented a gradient analysis of CCE and CKD for better understanding the effects on catastrophic forgetting, and have proposed ALI incorporating the merits of CCE and CKD.

Original: Second, we have proposed a feature replay scheme usingthe Cayley transform that requires less memory footprint than memorizing raw images.
Revised: Second, we have proposed a feature replay scheme using the Cayley transform that requires less memory footprint than memorizing raw images.

Original: Finally, wehave shown that ALIFE sets a new state of the art on standard ISS benchmarks.
Revised: Finally, we have shown that ALIFE sets a new state of the art on standard ISS benchmarks.

Revised: Acknowledgments and Disclosure of Funding

Revised: This work was supported by Institute of Information & Communications Technology Planing &

Revised: Evaluation (IITP) grant funded by the Korea government (MSIT)

Revised: (No.RS-2022-00143524, Development of Fundamental Technology and Integrated Solution for Next-Generation Automatic Artiﬁcial

Revised: Intelligence System) and the Yonsei Signature Research Cluster Program of 2022 (2022-22-0002).