Original: P ROGRESSIVE V ORONOI D IAGRAM S UBDIVISION E NABLES A CCURATE D ATA - FREE C LASSI NCREMENTAL L EARNING
Revised: P ROGRESSIVE V ORONOI D IAGRAM S UBDIVISION E NABLES A CCURATE D ATA - FREE C LASSI NCREMENTAL L EARNING

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Data-free Class-incremental Learning (CIL) is a challenging problem because rehearsing data from previous phases is strictly prohibited, causing catastrophic forgetting of Deep Neural Networks (DNNs).
Revised: Data-free Class-incremental Learning (CIL) is a challenging problem because rehearsing data from previous phases is strictly prohibited, causing catastrophic forgetting of Deep Neural Networks (DNNs).

Original: In this paper, we present iVoro , a novel framework derived from computational geometry.
Revised: In this paper, we present iVoro , a novel framework derived from computational geometry.

Original: We found Voronoi
Revised: We found Voronoi

Original: Diagram (VD), a classical model for space subdivision, is especially powerful for solving the CIL problem, because VD itself can be constructed favorably in an incremental manner – the newly added sites (classes) will only affect the proximate classes, making the non-contiguous classes hardly forgettable.
Revised: Diagram (VD), a classical model for space subdivision, is especially powerful for solving the CIL problem, because VD itself can be constructed favorably in an incremental manner – the newly added sites (classes) will only affect the proximate classes, making the non-contiguous classes hardly forgettable.

Original: Furthermore, we bridge DNN and VD using Power Diagram Reduction, and show that the VD structure can be progressively reﬁned along the phases using a divide-and-conquer algorithm.
Revised: Furthermore, we bridge DNN and VD using Power Diagram Reduction, and show that the VD structure can be progressively refined along the phases using a divide-and-conquer algorithm.

Original: Moreover, our VD construction is not restricted to the deep feature space, but is also applicable to multiple intermediate feature spaces, promoting
Revised: Moreover, our VD construction is not restricted to the deep feature space, but is also applicable to multiple intermediate feature spaces, promoting

Original: VD to be multilayer VD that efﬁciently captures multi-grained features from DNN.
Revised: VD to be multilayer VD that efficiently captures multi-grained features from DNN.

Original: Importantly, iVoro is also capable of handling uncertainty-aware test-time Voronoi cell assignment and has exhibited high correlations between geometric uncertainty and predictive accuracy (up to ∼ 0 . 9 ).
Revised: Importantly, iVoro is also capable of handling uncertainty-aware test-time Voronoi cell assignment and has exhibited high correlations between geometric uncertainty and predictive accuracy (up to ∼ 0 . 9 ).

Original: Putting everything together, iVoro achieves up to 25 .
Revised: Putting everything together, iVoro achieves up to 25 .

Original: 26% , 37 . 09% , and
Revised: 26% , 37 . 09% , and

Original: 21% improvements on CIFAR-100, TinyImageNet, and ImageNet-Subset, respectively, compared to the state-of-the-art non-exemplar
Revised: 21% improvements on CIFAR-100, TinyImageNet, and ImageNet-Subset, respectively, compared to the state-of-the-art non-exemplar

Original: CIL approaches.
Revised: CIL approaches.

Original: In conclusion, iVoro enables highly accurate, privacy-preserving, and geometrically interpretable CIL that is particularly useful when cross-phase data sharing is forbidden, e.g. in medical applications.
Revised: In conclusion, iVoro enables highly accurate, privacy-preserving, and geometrically interpretable CIL that is particularly useful when cross-phase data sharing is forbidden, e.g. in medical applications.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: [Figure]
Revised: [Figure]

Original: sites (A), and when a new site (B) or a clique of new sites (C) is added to the system.

Original: In many real-world applications such as medical imaging-based diagnosis, the learning system is usually required to be expandable to new classes, for example, from common to rare inherited retinal diseases (IRDs) (Miere et al., 2020), or from coarse to ﬁne chest radiographic ﬁndings (Syeda-Mahmood et al., 2020), and importantly, without losing the knowledge already learned.
Revised: In many real-world applications such as medical imaging-based diagnosis, the learning system is usually required to be expandable to new classes, for example, from common to rare inherited retinal diseases (IRDs) (Miere et al., 2020), or from coarse to fine chest radiographic findings (Syeda-Mahmood et al., 2020), and importantly, without losing the knowledge already learned.

Original: This motivates the concept of incremental learning (IL)
Revised: This motivates the concept of incremental learning (IL)

Original: (Hou et al., 2019; Wu et al., 2019; Zhu et al., 2021;
Revised: (Hou et al., 2019; Wu et al., 2019; Zhu et al., 2021;

Original: Liu et al., 2021b), also known as continual learning (Parisi et al., 2019; Delange et al., 2021; Chaudhry et al., 2019), which has drawn growing interest in recent years.
Revised: Liu et al., 2021b), also known as continual learning (Parisi et al., 2019; Delange et al., 2021; Chaudhry et al., 2019), which has drawn growing interest in recent years.

Original: Although Deep Neural Networks (DNNs) have become the de facto method of choice due to their extraordinary ability to learn from complex data, they still suffer from severe catastrophic forgetting (McCloskey & Cohen, 1989;
Revised: Although Deep Neural Networks (DNNs) have become the de facto method of choice due to their extraordinary ability to learn from complex data, they still suffer from severe catastrophic forgetting (McCloskey & Cohen, 1989;

Original: Goodfellow et al., 2014; Kemker et al., 2018) when adapting to new tasks that contain only unseen training samples from novel classes.
Revised: Goodfellow et al., 2014; Kemker et al., 2018) when adapting to new tasks that contain only unseen training samples from novel classes.

Original: To mitigate this issue, Rebufﬁ et al.
Revised: To mitigate this issue, Rebuffi et al.

Original: (2017) proposed the paradigm of memory-based class-incremental learning (CIL) (Belouadah & Popescu, 2019; Zhao et al., 2020; Hou et al., 2019; Castro et al., 2018;
Revised: (2017) proposed the paradigm of memory-based class-incremental learning (CIL) (Belouadah & Popescu, 2019; Zhao et al., 2020; Hou et al., 2019; Castro et al., 2018;

Original: Wu et al., 2019; Liu et al., 2021a; 2020a; 2021b) in which a small portion of samples (e.g.,exemplars per class) will be stored to use in the subsequent phases.
Revised: Wu et al., 2019; Liu et al., 2021a; 2020a; 2021b) in which a small portion of samples (e.g.,exemplars per class) will be stored to use in the subsequent phases.

Original: However, the storing and sharing of data, e.g. medical images, may not be feasible due to privacy considerations.
Revised: However, the storing and sharing of data, e.g. medical images, may not be feasible due to privacy considerations.

Original: Another line of methods memorize (part of) network and increase the model capacity for new classes (Rusu et al., 2016; Li et al., 2019; Wang et al., 2017; Yoon et al., 2017), which may incur unbounded memory consumption for long task sequence.
Revised: Another line of methods memorize (part of) network and increase the model capacity for new classes (Rusu et al., 2016; Li et al., 2019; Wang et al., 2017; Yoon et al., 2017), which may incur unbounded memory consumption for long task sequence.

Original: Hence, in this paper, we focus on the challenging data-free CIL problem under the strictest memory and privacy constraints – no stored exemplars and ﬁxed model capacity.
Revised: Hence, in this paper, we focus on the challenging data-free CIL problem under the strictest memory and privacy constraints – no stored exemplars and fixed model capacity.

Original: Despite extensive research in recent years (see Appendix A for a literature review), three challenges still pose an obstacle to successful CIL.
Revised: Despite extensive research in recent years (see Appendix A for a literature review), three challenges still pose an obstacle to successful CIL.

Original: (I)
Revised: (I)

Original: During the course of isolated training on new data, the feature distributions of the old classes are usually dramatically changed (see Fig.
Revised: During the course of isolated training on new data, the feature distributions of the old classes are usually dramatically changed (see Fig.

Original: 2 (A) for an illustration).
Revised: 2 (A) for an illustration).

Original: Knowledge Distillation (KD) (Hinton et al., 2015) has become a routine in many CIL methods (Li & Hoiem, 2017;
Revised: Knowledge Distillation (KD) (Hinton et al., 2015) has become a routine in many CIL methods (Li & Hoiem, 2017;

Original: Schwarz et al., 2018; Castro et al., 2018;
Revised: Schwarz et al., 2018; Castro et al., 2018;

Original: Hou et al., 2019; Dhar et al., 2019; Douillard et al., 2020; Zhu et al., 2021) to partially maintain the spatial distribution of old classes.
Revised: Hou et al., 2019; Dhar et al., 2019; Douillard et al., 2020; Zhu et al., 2021) to partially maintain the spatial distribution of old classes.

Original: The KD loss, however, is typically applied onto the whole network, and a strong KD loss may potentially degenerate the network’s ability to adapt to novel classes.
Revised: The KD loss, however, is typically applied onto the whole network, and a strong KD loss may potentially degenerate the network’s ability to adapt to novel classes.

Original: (II)
Revised: (II)

Original: Without the full access to old data, the decision boundaries cannot be learned precisely, making it harder to discriminate between old and new classes.
Revised: Without the full access to old data, the decision boundaries cannot be learned precisely, making it harder to discriminate between old and new classes.

Original: Taking inspiration from metric-based Few-shot Learning (FSL) (Snell et al., 2017), PASS (Zhu et al., 2021) memorizes a set of prototypes (feature centroids) and generates features augmented by Gaussian noise for a joint training in new phases.
Revised: Taking inspiration from metric-based Few-shot Learning (FSL) (Snell et al., 2017), PASS (Zhu et al., 2021) memorizes a set of prototypes (feature centroids) and generates features augmented by Gaussian noise for a joint training in new phases.

Original: However, feature centroids might be suboptimal to represent the whole class, which is not necessarily normally distributed (Fig. 2 (B)).
Revised: However, feature centroids might be suboptimal to represent the whole class, which is not necessarily normally distributed (Fig. 2 (B)).

Original: (III)
Revised: (III)

Original: Since the old classes and the new classes are learned in a disjoint manner, their distributions are likely to be overlapped, which becomes even severer in our exemplar-free setting as the old data is totally absent.
Revised: Since the old classes and the new classes are learned in a disjoint manner, their distributions are likely to be overlapped, which becomes even severer in our exemplar-free setting as the old data is totally absent.

Original: To circumvent this issue, Task-incremental learning (TIL) (Shin et al., 2017; Kirkpatrick et al., 2017; Zenke et al., 2017; Wu et al., 2018; Lopez-Paz & Ranzato, 2017;
Revised: To circumvent this issue, Task-incremental learning (TIL) (Shin et al., 2017; Kirkpatrick et al., 2017; Zenke et al., 2017; Wu et al., 2018; Lopez-Paz & Ranzato, 2017;

Original: Buzzega et al., 2020; Cha et al., 2021; Pham et al., 2021; Fernando et al., 2017) assumes the phase within which a class was learned is known, which is generally unrealistic in practice.
Revised: Buzzega et al., 2020; Cha et al., 2021; Pham et al., 2021; Fernando et al., 2017) assumes the phase within which a class was learned is known, which is generally unrealistic in practice.

Original: CIL is not grounded on this assumption.
Revised: CIL is not grounded on this assumption.

Original: In this paper, we tackle the CIL problem from a geometric point of view.
Revised: In this paper, we tackle the CIL problem from a geometric point of view.

Original: Voronoi Diagram (VD) is a classical model for space subdivision and is the underlying geometric structure of the 1-nearest neighbor classiﬁer (Lee, 1982).
Revised: Voronoi Diagram (VD) is a classical model for space subdivision and is the underlying geometric structure of the 1-nearest neighbor classifier (Lee, 1982).

Original: We ﬁnd that VD bears a close analogy to incremental learning, because VD itself can be constructed favorably in an incremental manner – the newly added sites (classes) will roughly change only the cells of the neighboring classes, making the non-contiguous classes untouched and thus hardly forgattable (see Figure 1).
Revised: We find that VD bears a close analogy to incremental learning, because VD itself can be constructed favorably in an incremental manner – the newly added sites (classes) will roughly change only the cells of the neighboring classes, making the non-contiguous classes untouched and thus hardly forgattable (see Figure 1).

Original: Based on this intuition, in this paper, we present a holistic geometric framework based on VD that signiﬁcantly surmounts all the listed obstacles.
Revised: Based on this intuition, in this paper, we present a holistic geometric framework based on VD that significantly surmounts all the listed obstacles.

Original: The contributions can be summarized as follows:
Revised: The contributions can be summarized as follows:

Original: [Figure]
Revised: [Figure]

Original: We explore, for the ﬁrst time, the idea of using prototypical networks (Snell et al., 2017) for CIL, which is equivalent to constructing a VD in the (ﬁxed) feature space (denoted as iVoro).
Revised: We explore, for the first time, the idea of using prototypical networks (Snell et al., 2017) for CIL, which is equivalent to constructing a VD in the (fixed) feature space (denoted as iVoro).

Original: We show that the within-phase boundaries of
Revised: We show that the within-phase boundaries of

Original: VD can be progressively reﬁned using a divide-andconquer algorithm (iVoro-D).
Revised: VD can be progressively refined using a divide-andconquer algorithm (iVoro-D).

Original: When it comes to test-time Voronoi cell assignment, we devise two protocols, augmentation consensus (iVoro-AC) and integration (iVoro-AI), for the postprocessing of Self-supervised Learning (SSL)-based label augmentation, with quantitative uncertainty awareness.
Revised: When it comes to test-time Voronoi cell assignment, we devise two protocols, augmentation consensus (iVoro-AC) and integration (iVoro-AI), for the postprocessing of Self-supervised Learning (SSL)-based label augmentation, with quantitative uncertainty awareness.

Original: Finally, we introduce multilayer features to build a multilayer VD, which consistently enhances the performance (iVoro-L).
Revised: Finally, we introduce multilayer features to build a multilayer VD, which consistently enhances the performance (iVoro-L).

Original: O UR M AIN I DEAS AND R ESULTS
Revised: Main Ideas and Results

Original: iVoro.
Revised: iVoro.

Original: We begin with the simplest scenario in which the feature extractor is frozen after the ﬁrst phase, and the prototypes ( { c } ) are used to construct VD.
Revised: We begin with the simplest scenario in which the feature extractor is frozen after the first phase, and the prototypes ( { c } ) are used to construct VD.

Original: iVoro-D. iVoro treats all prototypes equally regardless of at which phase they present, and determines
Revised: iVoro-D. iVoro treats all prototypes equally regardless of at which phase they present, and determines

Original: Voronoi boundaries all by bisecting prototypes.
Revised: Voronoi boundaries all by bisecting prototypes.

Original: However, without considering data distribution, the bisector of two prototypes is not optimal especially within a certain phase.
Revised: However, without considering data distribution, the bisector of two prototypes is not optimal especially within a certain phase.

Original: We establish an explicit connection between DNN and VD using Voronoi Diagram Reduction (Ma et al., 2022a) and show that the within-phase disicion boundaries (induced by { ˜ c } ) can be reﬁned by DNN (i.e. linear probing) and be aggregated into the global VD by a divide-and-conquer (D&C) algorithm (iVoro-D).
Revised: We establish an explicit connection between DNN and VD using Voronoi Diagram Reduction (Ma et al., 2022a) and show that the within-phase disicion boundaries (induced by { ˜ c } ) can be refined by DNN (i.e. linear probing) and be aggregated into the global VD by a divide-and-conquer (D&C) algorithm (iVoro-D).

Original: iVoro-AC/AI.
Revised: iVoro-AC/AI.

Original: Geometrically, SSL-based label augmentation (Lee et al., 2020) will duplicate one
Revised: Geometrically, SSL-based label augmentation (Lee et al., 2020) will duplicate one

Original: Voronoi cell to be multiple (possibly disjoint)
Revised: Voronoi cell to be multiple (possibly disjoint)

Original: Voronoi cells (see Fig.
Revised: Voronoi cells (see Fig.

Original: (D)), and this will cause ambiguity when assigning a query example to a cell, suggesting that uncertainty quantiﬁcation cannot be neglected in test-time.
Revised: (D)), and this will cause ambiguity when assigning a query example to a cell, suggesting that uncertainty quantification cannot be neglected in test-time.

Original: Here we propose two protocols to resolve this ambiguity, namely, augmentation consensus (iVoro-AC) and augmentation integration (iVoro-AI).
Revised: Here we propose two protocols to resolve this ambiguity, namely, augmentation consensus (iVoro-AC) and augmentation integration (iVoro-AI).

Original: We also show that the entropy-based geometric variance (Ding & Xu, 2020) is a good indicator of the uncertainty of this assignment, with high Pearson correlation coefﬁcients up to ∼ 0 .
Revised: We also show that the entropy-based geometric variance (Ding & Xu, 2020) is a good indicator of the uncertainty of this assignment, with high Pearson correlation coefficients up to ∼ 0 .

Original: iVoro-L. Until now, only deep features from the last layer are used for VD construction.
Revised: iVoro-L. Until now, only deep features from the last layer are used for VD construction.

Original: However, the intermediate feature could also be informative to aid the VD construction.
Revised: However, the intermediate feature could also be informative to aid the VD construction.

Original: Cluster-induced Voronoi
Revised: Cluster-induced Voronoi

Original: Diagram (CIVD)
Revised: Diagram (CIVD)

Original: (Chen et al., 2013; 2017; Huang & Xu, 2020; Huang et al., 2021), which allows for multiple centers per Voronoi cell, has recently achieved remarkable success in metric-based FSL by incorporating heterogeneous features to VD (Ma et al., 2022a).
Revised: (Chen et al., 2013; 2017; Huang & Xu, 2020; Huang et al., 2021), which allows for multiple centers per Voronoi cell, has recently achieved remarkable success in metric-based FSL by incorporating heterogeneous features to VD (Ma et al., 2022a).

Original: As a matter of fact, for a deep neural network, the feature induced by every layer can all be used to construct a VD.
Revised: As a matter of fact, for a deep neural network, the feature induced by every layer can all be used to construct a VD.

Original: Finally, we also explore the idea to build a multilayer VD by using features elicited from multiple blocks of DNN.
Revised: Finally, we also explore the idea to build a multilayer VD by using features elicited from multiple blocks of DNN.

Original: Broader impact.
Revised: Broader impact.

Original: The fully-ﬂedged iVoro achieves up to 25 .
Revised: The fully-fledged iVoro achieves up to 25 .

Original: 26% , 37 . 09% , and
Revised: 26% , 37 . 09% , and

Original: 21% improvements on CIFAR-100, TinyImageNet, and ImageNet-Subset, respectively, compared with the state-of-the-art non-exemplar CIL approaches.
Revised: 21% improvements on CIFAR-100, TinyImageNet, and ImageNet-Subset, respectively, compared with the state-of-the-art non-exemplar CIL approaches.

Original: Based on the frozen model trained at the ﬁrst phase, iVoro and all its variants incur no additional training burden, and at the same time preserve the privacy of data from previous phases.
Revised: Based on the frozen model trained at the first phase, iVoro and all its variants incur no additional training burden, and at the same time preserve the privacy of data from previous phases.

Original: It is worth noting that, although iVoro focuses on exemplar-free CIL, it outperforms even all the exemplar-based CIL methods.
Revised: It is worth noting that, although iVoro focuses on exemplar-free CIL, it outperforms even all the exemplar-based CIL methods.

Original: We believe iVoro could be further boosted when a small number of exemplars are allowed, which we leave for future work.
Revised: We believe iVoro could be further boosted when a small number of exemplars are allowed, which we leave for future work.

Original:  M ETHODOLOGY 3. P RELIMINARIES : C LASS -I NCREMENTAL L EARNING
Revised:  M ETHODOLOGY 2. P RELIMINARIES : C LASS -I NCREMENTAL L EARNING

Original: In CIL, the data comes as a stream and a single model is trained on current data locally without revisiting previous data, but should ideally be able to discriminate between all classes it has seen so far.
Revised: In CIL, the data comes as a stream and a single model is trained on current data locally without revisiting previous data, but should ideally be able to discriminate between all classes it has seen so far.

Original: Speciﬁcally, let D =
Revised: Specifically, let D =

Original: {D t } T t =1 be the data stream in which D t 
Revised: {D t } Tt =1 be the data stream in which D t = { ( x t,i , y t,i ) } N t i

Original: = { ( x t,i , y t,i ) } N t i =1 is the dataset at time step t , with data x t,i ∈ D and label y t,i ∈ C t .
Revised: =1 is the dataset at time step t , with data x t,i ∈ D and label y t,i ∈ C t .

Original: D is an arbitrary domain, e.g., natural image, and C t is the set of classes at phase t .
Revised: D is an arbitrary domain, e.g., natural image, and C t is the set of classes at phase t .

Original: The dataset D t contains N t,k , k ∈ { 1 , ..., K t } samples for the
Revised: The dataset D t contains N t,k , k ∈ { 1 , ..., K t } samples for the

Original: K t classes (i.e. N t = (cid:80) K t k =1 N t,k ).
Revised: K t classes (i.e. N t = (cid:80) K t k =1 N t,k ).

Original: Notice that C i , C j for two arbitrary phases i, j are disjoint, i.e.
Revised: Notice that C i , C j for two arbitrary phases i, j are disjoint, i.e.

Original: C i ∩C
Revised: C i ∩C

Original: j = ∅ , ∀ i, j : i(cid:54) =
Revised: j = ∅ , ∀ i, j : i

Revised: ̸

Original:  j .
Revised: = j .

Original: The uniﬁed model consists of a feature extractor φ and a classiﬁcation head θ .
Revised: The unified model consists of a feature extractor ϕ and a classification head θ .

Original: The feature extractor is a deep neural network z = φ ( x ) , z ∈ R n that maps from image domain D to feature domain R n , and is (traditionally) trained continuously at each phase t .
Revised: The feature extractor is a deep neural network z = ϕ ( x ) , z ∈ R n that maps from image domain D to feature domain R n , and is (traditionally) trained continuously at each phase t .

Original: In this section,
Revised: In this section,

Original: T , t , and τ denote total phase, current phase, and historical phase, respectively (i.e. t ∈ { 1 , ..., T } , τ ∈ { 1 , ..., t } ).
Revised: T , t , and τ denote total phase, current phase, and historical phase, respectively (i.e. t ∈ { 1 , ..., T } , τ ∈ { 1 , ..., t } ).

Original: 3. C ONSTRUCTING V ORONOI D IAGRAMS : A F EATURE E XTRACTOR IS A LL Y OU N EED
Revised: 2. C ONSTRUCTING V ORONOI D IAGRAMS : A F EATURE E XTRACTOR IS A LL Y OU N EED

Original: In many CIL methods, the feature extractor φ and classiﬁcation head θ are jointly and continuously optimized during every phase t guided by carefully designed losses (Zhu et al., 2021).
Revised: In many CIL methods, the feature extractor ϕ and classification head θ are jointly and continuously optimized during every phase t guided by carefully designed losses (Zhu et al., 2021).

Original: As a starting point, in this section, we freeze the feature extractor φ after the ﬁrst phase and use a Voronoi Diagram (i.e., a 1-nearest-neighbor classiﬁers) to be θ as an extremely simple baseline method (denoted as iVoro), upon which we will then gradually add component introduced in Sec. 1.
Revised: As a starting point, in this section, we freeze the feature extractor ϕ after the first phase and use a Voronoi Diagram (i.e., a 1-nearest-neighbor classifiers) to be θ as an extremely simple baseline method (denoted as iVoro), upon which we will then gradually add component introduced in Sec. 1.

Original: First, we introduce
Revised: First, we introduce

Original: Power Diagram (PD), a generalized version of VD:
Revised: Power Diagram (PD), a generalized version of VD:

Original: Deﬁnition 3.1 (Power Diagram (Aurenhammer, 1987) and Voronoi Diagram) .
Revised: Definition 2.1 (Power Diagram (Aurenhammer, 1987) and Voronoi Diagram) .

Original: Let Ω = { ω 1 , ..., ω K } be a partition of the space R n , and C = { c 1 , ..., c K } be a set of centers (also called sites ) such that ∪ Kr =1 ω r = R n , ∩ Kr =1 ω r = ∅ .
Revised: Let Ω = { ω 1 , ..., ω K } be a partition of the space R n , and C = { c 1 , ..., c K } be a set of centers (also called sites ) such that ∪ Kr =1 ω r = R n , ∩ Kr =1 ω r = ∅ .

Original: In addition, each center is associated with a weight ν r ∈ { ν 1 , ..., ν K } ⊆ R + .
Revised: In addition, each center is associated with a weight ν r ∈ { ν 1 , ..., ν K } ⊆ R + .

Original: Then, the set of pairs { ( ω 1 , c 1 , ν 1 ) , ..., ( ω K , c L , ν K ) } is a Power Diagram (PD), where each cell is obtained via ω r = { z ∈ R n : r ( z ) =
Revised: Then, the set of pairs { ( ω 1 , c 1 , ν 1 ) , ..., ( ω K , c L , ν K ) } is a Power Diagram (PD), where each cell is obtained via ω r = { z ∈ R n : r ( z ) =

Original: r } , r ∈ { 1 , .., K } , with r ( z ) =
Revised: r } , r ∈ { 1 , .., K } , with r ( z ) =

Original: arg min k ∈{ 1 ,...,K } d ( z , c k ) 2 − ν k .
Revised: arg min k ∈{ 1 ,...,K } d ( z , c k ) 2 − ν k .

Original: If the weights are equal for all k , i.e. ν k = ν k (cid:48) , ∀ k, k (cid:48) ∈ { 1 , ..., K } , then a PD collapses to a Voronoi Diagram (VD).
Revised: If the weights are equal for all k , i.e. ν k = ν k ′ , ∀ k, k ′ ∈ { 1 , ..., K } , then a PD collapses to a Voronoi Diagram (VD).

Original: Prototypes.
Revised: Prototypes.

Original: As a baseline model, the class centers for iVoro are simply chosen to be the prototypes (feature mean of one class): c τ,k =1 N τ,k (cid:80) i ∈{ 1 ,...,N τ,k } ,y =
Revised: As a baseline model, the class centers for iVoro are simply chosen to be the prototypes (feature mean of one class): c τ,k =

Original: k φ ( x τ,i ) , ν τ,k = 0 , τ ∈ { 1 , ..., t } , k ∈ { 1 , ..., K τ } .
Revised: [Equation] { 1 , ..., K τ } .

Original: We name those centers prototypical centers .
Revised: We name those centers prototypical centers .

Original: Note that this set of centers { c τ,k } carries prototypes for all classes, old and new, up to time t .
Revised: Note that this set of centers { c τ,k } carries prototypes for all classes, old and new, up to time t .

Original: In test-time, a query sample x is assigned to the nearest class
Revised: In test-time, a query sample x is assigned to the nearest class

Original: ˆ y =
Revised: ˆ y =

Original: C τ (cid:48) ,k (cid:48) s.t.
Revised: C τ ′ ,k ′ s.t.

Original: d ( z , c τ (cid:48) ,k (cid:48) )
Revised: d ( z , c τ ′ ,k ′ )

Original: =
Revised: =

Original: min τ,k d ( z , c τ,k ) in which d ( z , c τ,k ) =
Revised: min τ,k d ( z , c τ,k ) in which d ( z , c τ,k ) =

Original: ||
Revised: ||

Original: z − c τ,k || 22 .
Revised: z − c τ,k || 22 .

Original: Parameterized Feature Transformation.
Revised: Parameterized Feature Transformation.

Original: Although PASS (Zhu et al., 2021) uses Gaussian noise to augment the data, the actual features are not necessarily normally distributed.
Revised: Although PASS (Zhu et al., 2021) uses Gaussian noise to augment the data, the actual features are not necessarily normally distributed.

Original: To encourage the normality of feature distribution here we adopt compositional feature transformation commonly used in FSL (Ma et al., 2022a): (1) L 2 normalization projects the feature onto the unit sphere: f ( z ) =
Revised: To encourage the normality of feature distribution here we adopt compositional feature transformation commonly used in FSL (Ma et al., 2022a): (1) L 2 normalization projects the feature onto the unit sphere: f ( z ) =

Original: z ||
Revised: z ||

Original: z
Revised: z

Original: || 2 ; (2) linear transformation performs the scaling and shifting: g w,η ( z ) =
Revised: || 2 ; (2) linear transformation performs the scaling and shifting: g w,η ( z ) =

Original: w z + η ; and (3) Tukey’s ladder of powers transformation further improves the Gaussianity: h λ ( z ) =
Revised: w z + η ; and (3) Tukey’s ladder of powers transformation further improves the Gaussianity: h λ ( z ) =

Original: (cid:26) z λ if λ (cid:54) = log( z ) if λ = 0 .
Revised: (cid:26) z λ if λ ̸ =log( z ) if λ = 0 .

Original: Finally, the feature transformation is the composition of three: ( h λ ◦ g w,η ◦ f )( z ) , parameterized by w, η, λ .
Revised: Finally, the feature transformation is the composition of three: ( h λ ◦ g w,η ◦ f )( z ) , parameterized by w, η, λ .

Original: If all features (for both training and testing set) go through this normalization function, then iVoro becomes iVoro-N.
Revised: If all features (for both training and testing set) go through this normalization function, then iVoro becomes iVoro-N.

Original: 3. D IVIDE AND C ONQUER : P ROGRESSIVE V ORONOI D IAGRAMS FOR CIL
Revised: 2. D IVIDE AND C ONQUER : P ROGRESSIVE V ORONOI D IAGRAMS FOR CIL

Original: As mentioned earlier, iVoro (and iVoro-N) treats all classes equally and separates them all by bisectors, regardless of at which phase they appear.
Revised: As mentioned earlier, iVoro (and iVoro-N) treats all classes equally and separates them all by bisectors, regardless of at which phase they appear.

Original: However, for two classes C τ,k 1 , C τ,k 2 appear in the same phase τ , we can in fact draw better boundary by training a linear probing model parametrized by
Revised: However, for two classes C τ,k 1 , C τ,k 2 appear in the same phase τ , we can in fact draw better boundary by training a linear probing model parametrized by

Original: W , b in the ﬁxed feature space.
Revised: W , b in the fixed feature space.

Original: After the training, the locating of new Voronoi center requires an explicit relationship between the probing model and VD.
Revised: After the training, the locating of new Voronoi center requires an explicit relationship between the probing model and VD.

Original: More formally, at phase t a linear classiﬁer with cross-entropy loss is optimized on the local data D t :
Revised: More formally, at phase t a linear classifier with cross-entropy loss is optimized on the local data D t :

Original: [Equation]
Revised: [Equation]

Original: in which W t,k , b t,k are the linear weight and bias for class C t,k .
Revised: in which W t,k , b t,k are the linear weight and bias for class C t,k .

Original: As a parameterized model, this linear probing can ideally improve the discrimination within C t .
Revised: As a parameterized model, this linear probing can ideally improve the discrimination within C t .

Original: However, it is still non-trivial to merge all { W τ,k , b τ,k } tτ =1 , since the task identity is not assumed to be known like in TIL.
Revised: However, it is still non-trivial to merge all { W τ,k , b τ,k } t τ =1 , since the task identity is not assumed to be known like in TIL.

Original: To solve this, we get geometric insight from (Ma et al., 2022a) which directly connects linear probing model and VD by the theorem shown as follows:
Revised: To solve this, we get geometric insight from (Ma et al., 2022a) which directly connects linear probing model and VD by the theorem shown as follows:

Original: Theorem 3.1 (Voronoi Diagram Reduction (Ma et al., 2022a)) .
Revised: Theorem 2.1 (Voronoi Diagram Reduction (Ma et al., 2022a)) .

Original: The linear classiﬁer parameterized by W , b partitions the input space R n to a Voronoi Diagram with centers { ˜ c 1 , ..., ˜ c K } given by ˜ c
Revised: The linear classifier parameterized by W , b partitions the input space R n to a Voronoi Diagram with centers { ˜ c 1 , ..., ˜ c K } given by ˜ c

Original: k = 12 W k if b k = − 

Revised: k = 12 W k if b k = −

Original: || W k
Revised: || W k

Original: || 22 , k = 1 , ..., K .
Revised: || 22 , k = 1 , ..., K .

Original: For completeness, we also include the proof in Appendix E.
Revised: For completeness, we also include the proof in Appendix E.

Original: During linear probing, if Thm. 3.1 is satisﬁed, then it is guaranteed that the resulting centers (referred to as probing-induced centers ) { ˜ c t,k } K t k =1 will also induce a VD (locally in phase t ).
Revised: During linear probing, if Thm. 2.1 is satisfied, then it is guaranteed that the resulting centers (referred to as probing-induced centers ) { ˜ c t,k } K t k =1 will also induce a VD (locally in phase t ).

Original: Now given that we have two sets of centers { c τ,k } and { ˜ c τ,k } , with the latter being better locally but are not transferable across phases, we devise a divide-and-conquer (D&C) algorithm that progressively construct the decision boundaries from the two sets of centers, boosting iVoro to iVoro-D.
Revised: Now given that we have two sets of centers { c τ,k } and { ˜ c τ,k } , with the latter being better locally but are not transferable across phases, we devise a divide-and-conquer (D&C) algorithm that progressively construct the decision boundaries from the two sets of centers, boosting iVoro to iVoro-D.

Original: Divide.
Revised: Divide.

Original: Fortunately, the total classes {C τ } tτ =1 have been split into already disjoint t cliques.
Revised: Fortunately, the total classes {C τ } tτ =1 have been split into already disjoint t cliques.

Original: Conquer.
Revised: Conquer.

Original: Within each clique (i.e. phase) τ , the boundary for any two classes C τ,k 1 , C τ,k 2 is the bisector separating the probing-induced centers ˜
Revised: Within each clique (i.e. phase) τ , the boundary for any two classes C τ,k 1 , C τ,k 2 is the bisector separating the probing-induced centers ˜

Original: c
Revised: c

Original: τ,k 1 , ˜ c
Revised: τ,k 1 , ˜ c

Original: τ,k 2 , denoted as Γ τ,k 1 ,τ,k 2 = { z (cid:48) ∈
Revised: τ,k 2 , denoted as Γ τ,k 1 ,τ,k 2 = { z ′ ∈

Original: R n | v T z (cid:48) − q = 0 } where v = 
Revised: R n | v T z ′ − q = 0 } where v = ˜ c [Equation] − [Equation]

Revised: 2 and q =

Original: [Equation]
Revised: [Equation]

Original: ˜
Revised: ˜

Original: [Equation]
Revised: [Equation]

Revised: 22 −

Revised: [Equation]

Original: and q =
Revised: ˜ [Equation] [Equation] − ˜ c

Original: [Equation]
Revised: [Equation]

Original: {+k 
 || 2 .+} When merging cliques τ 1 , τ 2 , we instead resort to the prototypical centers for space partition: for any c τ 1 ,k in clique τ 1 and any c τ 2 ,k (cid:48) in clique τ 2 , their bisector is Γ τ 1 ,k,τ 2 ,k (cid:48)
Revised: {+k 
 || 2 .+} When merging cliques τ 1 , τ 2 , we instead resort to the prototypical centers for space partition: for any c τ 1 ,k in clique τ 1 and any c τ 2 ,k ′ in clique τ 2 , their bisector is Γ τ 1 ,k,τ 2 ,k ′ =

Original: = { z [-(cid:
 )-] ∈ R n | v T z (cid:48) − q = 0 } where v = c τ 1 ,k − c τ 2 ,k (cid:48) ||
Revised: { z [-(cid:
 )-]′ ∈ R n | v T z ′ − q = 0 } where v = c τ 1 ,k − c τ 2 ,k ′ ||

Original: c τ 1 ,k − c τ 2 ,k (cid:48)
Revised: c τ 1 ,k − c τ 2 ,k ′

Original: || 2 and q = ||
Revised: || 2 and q = ||

Original: c τ 1 ,k
Revised: c τ 1 ,k

Original: || 22 −|| c τ 2 ,k (cid:48)
Revised: || 22 −|| c τ 2 ,k ′

Original: ||
Revised: ||

Original:  || c τ 1 ,k − c τ 2 ,k (cid:48)
Revised: 22 2 || c τ 1 ,k − c τ 2 ,k ′

Original: || 2 .
Revised: || 2 .

Original: In this way, the overall space partition would beneﬁt from both locally probing-induced VD and globally prototype-based VD.
Revised: In this way, the overall space partition would benefit from both locally probing-induced VD and globally prototype-based VD.

Original: See Fig.
Revised: See Fig.

Original: G.3and Fig. G.4 for an illustrative comparison of iVoro and iVoro-D.
Revised: G.3 for an illustrative comparison of iVoro and iVoro-D.

Original: Querying the VD.
Revised: Querying the VD.

Original: In test-time, one can ﬁnd the assigned Voronoi cell for query example x by eliminating one class in each round according to sign( v T z (cid:48) − q ) , starting from a randomly selected boundary, so the time complexity is O ( (cid:80) tτ
Revised: In test-time, one can find the assigned Voronoi cell for query example x by eliminating one class in each round according to sign( v T z ′ − q ) , starting from a randomly selected boundary, so the time complexity is O ( (cid:80) tτ

Original: =1 K τ ).
Revised: =1 K τ )

Original: 3. A UGMENTATION I NTEGRATION : U NCERTAINTY - AWARE T EST - TIME V ORONOI C ELL A SSIGNMENT
Revised: . 2. A UGMENTATION I NTEGRATION : U NCERTAINTY - AWARE T EST - TIME V ORONOI C ELL A SSIGNMENT

Original: Self-supervised Label Augmentation.
Revised: Self-supervised Label Augmentation.

Original: To enhance the discriminative power of CIL method, SSLbased label augmentation (Lee et al., 2020) has been used to expand the original K t classes to 4 K t by rotating the original image x .
Revised: To enhance the discriminative power of CIL method, SSLbased label augmentation (Lee et al., 2020) has been used to expand the original K t classes to 4 K t by rotating the original image x .

Original: Speciﬁcally, for image x , the rotated image x ( α ) =
Revised: Specifically, for image x , the rotated image x ( α ) =

Original: rotate ( x , π 2 α ) , α ∈ { 0 , 1 , 2 , 3 } will be assigned to one of the expanded classes ˆ y = k ( α ) , k ∈ { 1 , ..., K } , K ∈ (cid:80) tτ
Revised: rotate ( x , π 2 α ) , α ∈ { 0 , 1 , 2 , 3 } will be assigned to one of the expanded classes ˆ y = k ( α ) , k ∈ { 1 , ..., K } , K ∈ (cid:80) tτ

Original: =1 K τ .
Revised: =1 K τ .

Original: In training time, the model is trained on the expanded dataset; however, in testing time, each of the duplicated images { x ( α ) } α ∈{ 0 , 1 , 2 , 3 } could possibly be assigned to each of the expanded classes { k ( α ) } α ∈{ 0 , 1 , 2 , 3 } , so this ambiguity has to be resolved, which has not been considered in previous
Revised: In training time, the model is trained on the expanded dataset; however, in testing time, each of the duplicated images { x ( α ) } α ∈{ 0 , 1 , 2 , 3 } could possibly be assigned to each of the expanded classes { k ( α ) } α ∈{ 0 , 1 , 2 , 3 } , so this ambiguity has to be resolved, which has not been considered in previous

Original: CIL methods.
Revised: CIL methods.

Original: Augmentation Consensus.
Revised: Augmentation Consensus.

Original: Let d ( α,α (cid:48) )
Revised: Let d ( α,α ′ )

Original: ∈
Revised: ∈

Original: R K be a vector, each component of which denotes the distance from φ ( x ) to a class that φ has learned, i.e. d ( α,α (cid:48) ) k
Revised: R K be a vector, each component of which denotes the distance from ϕ ( x ) to a class that ϕ has learned, i.e. d ( α,α ′ ) k

Original: = d ( φ ( x ( α ) ) , c k ( α (cid:48) ) )
Revised: = d ( ϕ ( x ( α ) ) , c k ( α ′ ) )

Original: =
Revised: =

Original: || φ ( x ( α ) )
Revised: || ϕ ( x ( α ) )

Original: − c k ( α (cid:48) )
Revised: − c k ( α ′ )

Original: || 2 2 , k ∈ { 1 , ..., K } , α, α (cid:48) ∈ { 0 , 1 , 2 , 3 } .
Revised: || 22 , k ∈ { 1 , ..., K } , α, α ′ ∈ { 0 , 1 , 2 , 3 } .

Original: Then we want to ﬁnd a consensus ˆ k , with the maximum occurrence among the 4 × 4 predictions { arg min k d( α,α (cid:48) )
Revised: Then we want to find a consensus ˆ k , with the maximum occurrence among the 4 × 4 predictions { arg min k d

Original:  k } α,α (cid:48) ∈{ 0 , 1 , 2 , 3 } .
Revised: ( α,α ′ ) k } α,α ′ ∈{ 0 , 1 , 2 , 3 } .

Original: Using augmentation consensus in test-time, iVoro is then retroﬁtted to iVoro-AC.
Revised: Using augmentation consensus in test-time, iVoro is then retrofitted to iVoro-AC.

Original: Augmentation Integration.
Revised: Augmentation Integration.

Original: Using the consensus from the augmented samples should be more robust than the individual prediction arg min k d (0 , 0) k itself, but it has not considered the accumulated distance, so alternatively, we propose to integral over all predictions from augmented samples:
Revised: Using the consensus from the augmented samples should be more robust than the individual prediction arg min k d (0 , 0) k itself, but it has not considered the accumulated distance, so alternatively, we propose to integral over all predictions from augmented samples:

Original: [Equation]
Revised: [Equation]

Original: If augmentation integration is applied, then iVoro becomes iVoro-AI.
Revised: If augmentation integration is applied, then iVoro becomes iVoro-AI.

Original: Uncertainty Quantiﬁcation.
Revised: Uncertainty Quantification.

Original: Since in iVoro-AC and iVoro-AI, the augmented samples collaboratively contribute to the ﬁnal prediction, the quantitative uncertainty becomes non-negligible, this is because for some rotation-invariant classes, e.g. balls, the rotation operation makes less sense.
Revised: Since in iVoro-AC and iVoro-AI, the augmented samples collaboratively contribute to the final prediction, the quantitative uncertainty becomes non-negligible, this is because for some rotation-invariant classes, e.g. balls, the rotation operation makes less sense.

Original: Hence, when assigning a query sample x to the augmented 4 × Voronoi cells, an uncertainty quantiﬁcation method is needed.
Revised: Hence, when assigning a query sample x to the augmented 4 × Voronoi cells, an uncertainty quantification method is needed.

Original: Truth Discovery Ensemble (TDE)
Revised: Truth Discovery Ensemble (TDE)

Original: (Ma et al., 2021) is the state-of-the-art uncertainty calibration method for DNNs, which ﬁnds the consensus among ensemble members by the minimization of entropy-based geometric variance (HV).
Revised: (Ma et al., 2021) is the state-of-the-art uncertainty calibration method for DNNs, which finds the consensus among ensemble members by the minimization of entropy-based geometric variance (HV).

Original: Here, we only borrow HV as an indicator for the uncertainty of the 4 × 4 predictions, and refer the readers to Ma et al.
Revised: Here, we only borrow HV as an indicator for the uncertainty of the 4 × 4 predictions, and refer the readers to Ma et al.

Original: (2021) for more details about TDE.
Revised: (2021) for more details about TDE.

Original: Given the mean vector of the augmented predictions d ∗ = 116 (cid:80) α,α (cid:48) d ( α,α (cid:48) )
Revised: Given the mean vector of the augmented predictions d ∗ = 116 (cid:80) α,α ′ d ( α,α ′ )

Original: ∈ R K , let V denote the total squared distance to d ∗ (i.e., V = (cid:80) α (cid:80)α (cid:48)
Revised: ∈ R K , let V denote the total squared distance to d ∗ (i.e., V = (cid:80) α (cid:80)

Original:  ||
Revised: α ′ ||

Original: d ∗ − d ( α,α (cid:48) )
Revised: d ∗ − d ( α,α ′ )

Original: || 2 ) and q ( α,α (cid:48) ) denotes the contribution of each d ( α,α (cid:48) ) to V (i.e., q ( α,α (cid:48) )
Revised: || 2 ) and q ( α,α ′ ) denotes the contribution of each d ( α,α ′ ) to V (i.e., q ( α,α ′ )

Original: =
Revised: =

Original: || d ∗ − d ( α,α (cid:48) )
Revised: || d ∗ − d ( α,α ′ )

Original: || 2 /V ).
Revised: || 2 /V ).

Original: Then the entropy induced by { q ( α,α (cid:48) ) } is:
Revised: Then the entropy induced by { q ( α,α ′ ) } is:

Original: [Equation]
Revised: [Equation]

Original: Based on these, we can deﬁne the HV as follows:
Revised: Based on these, we can define the HV as follows:

Original: Deﬁnition 3.2 (Entropy-based Geometric Variance (Ding & Xu, 2020)) .
Revised: Definition 2.2 (Entropy-based Geometric Variance (Ding & Xu, 2020)) .

Original: Given the point set { d ( α,α (cid:48) ) 
Revised: Given the point set { d ( α,α ′ ) } ⊆

Original: } ⊆ R K and a point d ∗ , the entropy based geometric variance (HV) is H × V where
Revised: R K and a point d ∗ , the entropy based geometric variance (HV) is H × V where

Original: H and V are deﬁned as shown above.
Revised: H and V are defined as shown above.

Original: For every query example x , we calculate HV ( x ) based on its { d ( α,α (cid:48) )
Revised: For every query example x , we calculate HV ( x ) based on its { d ( α,α ′ )

Original: } α,α (cid:48) ∈{ 0 , 1 , 2 , 3 } .
Revised: } α,α ′ ∈{ 0 , 1 , 2 , 3 } .

Original: Later we will show how HV could favorably indicate the uncertainty of the augmented prediction, and tell us when augmentation integration is useful.
Revised: Later we will show how HV could favorably indicate the uncertainty of the augmented prediction, and tell us when augmentation integration is useful.

Original: 3. M ULTILAYER V ORONOI D IAGRAMS
Revised: [Equation] [Figure]

Original: Block
Revised: Block

Original: ResNetBlock
Revised: ResNetBlock

Original: Block
Revised: Block

Original: Block{c k ⑴ } k {c k ⑵ } k {c k ⑶ } k {{c k(i) } Li=1 } k ∈ {1,...,K}
Revised: Block{c k ⑴ } k {c k ⑵ } k {c k ⑶ } k {{c k(i) } Li=1 } k ∈ {1,...,K}

Original: Cluster-induced
Revised: Cluster-induced

Original: Voronoi Diagram [44] iVoro-L
Revised: Voronoi Diagram [44] iVoro-L

Original: VD VD
Revised: VD VD

Original: VD
Revised: VD

Original: Until now, our VD construction is restricted to the deep feature space, i.e., x (cid:55)→ φ ( x ) ∈ R n .
Revised: Until now, our VD construction is restricted to the deep feature space, i.e., x (cid:55)→ ϕ ( x ) ∈ R n .

Original: However, the intermediate layers also contain information that supplementary to the ﬁnal layer and can be useful to our
Revised: However, the intermediate layers also contain information that supplementary to the final layer and can be useful to our

Original: VD construction.
Revised: VD construction.

Original: And this requires the integration of multiple VDs.
Revised: And this requires the integration of multiple VDs.

Original: Recently, Cluster-induced Voronoi Diagram (CIVD)
Revised: Recently, Cluster-induced Voronoi Diagram (CIVD)

Original: (Chen et al., 2017; Huang et al., 2021) and Cluster-to-cluster Voronoi Diagram (CCVD)
Revised: (Chen et al., 2017; Huang et al., 2021) and Cluster-to-cluster Voronoi Diagram (CCVD)

Original: (Ma et al., 2022a), two advanced VD structures, have shown remarkable ability to integrate multiple sets of centers for VD construction and achieve state-of-the-art performance in metric-based FSL.
Revised: (Ma et al., 2022a), two advanced VD structures, have shown remarkable ability to integrate multiple sets of centers for VD construction and achieve state-of-the-art performance in metric-based FSL.

Original: In this paper, we utilize the concept of CCVD for the integration of multiple VDs induced by multiple layers.
Revised: In this paper, we utilize the concept of CCVD for the integration of multiple VDs induced by multiple layers.

Original: We refer the readers to Ma et al.
Revised: We refer the readers to Ma et al.

Original: (2022a) for more details about CIVD/CCVD.
Revised: (2022a) for more details about CIVD/CCVD.

Original: Deﬁnition 3.3 (Cluster-to-cluster Voronoi Diagram) .
Revised: Definition 2.3 (Cluster-to-cluster Voronoi Diagram) .

Original: Let Ω = { ω 1 , ..., ω K } be a partition of the space R n , and
Revised: Let Ω = { ω 1 , ..., ω K } be a partition of the space R n , and

Original: C =
Revised: C =

Original: {C 1 , ..., C K } be a set of totally ordered sets with the same cardinality L (i.e.
Revised: {C 1 , ..., C K } be a set of totally ordered sets with the same cardinality L (i.e.

Original: |C 1 | =
Revised: |C 1 | =

Original: |C 2 | = ...
Revised: |C 2 | = ...

Original: =
Revised: =

Original: |C K | =
Revised: |C K | =

Original: L ).
Revised: L ).

Original: The set of pairs { ( ω 1 , C 1 ) , ..., ( ω K , C K ) } is a Cluster-to-cluster
Revised: The set of pairs { ( ω 1 , C 1 ) , ..., ( ω K , C K ) } is a Cluster-to-cluster

Original: Voronoi Diagram (CCVD) with respect to an inﬂuence function F ( C k , C ( z )) , and each cell is obtained via ω r = { z ∈ R n : r ( z ) = r } , r ∈ { 1 , .., K } , with r ( z ) =
Revised: Voronoi Diagram (CCVD) with respect to an influence function F ( C k , C ( z )) , and each cell is obtained via ω r = { z ∈ R n : r ( z ) = r } , r ∈ { 1 , .., K } , with r ( z ) =

Original: arg max k ∈{ 1 ,...,K } F ( C k , C ( z ))
Revised: arg max k ∈{ 1 ,...,K } F ( C k , C ( z ))

Original: where
Revised: where

Original: C ( z ) is the cluster (also a totally ordered set with cardinality L ) that query point z belongs to, meaning that, all points in this cluster (query cluster) will be assigned to the same cell.
Revised: C ( z ) is the cluster (also a totally ordered set with cardinality L ) that query point z belongs to, meaning that, all points in this cluster (query cluster) will be assigned to the same cell.

Original: The
Revised: The

Original: Inﬂuence Function is deﬁned upon two totally ordered sets C k
Revised: Influence Function is defined upon two totally ordered sets C k

Original: = { c ( i ) k } Li =1 and C ( z ) =
Revised: = { c ( i ) k } Li =1 and C ( z ) =

Original: { z ( i ) } Li =1 :
Revised: { z ( i ) } Li =1 :

Original: F ( C k , C ( z ))
Revised: F ( C k , C ( z ))

Original: = − sign( γ ) (cid:80) Li =

Revised: = − sign( γ ) (cid:80) Li =

Original: d ( c ( i ) k , z
Revised: d ( c ( i ) k , z

Original: ( i ) ) γ .
Revised: ( i ) ) γ .

Original: As CCVD is a ﬂexible framework and can be applied to iVoro-D/AC/AI, here, as an example, we show how CCVD can be use to boost iVoro.
Revised: As CCVD is a flexible framework and can be applied to iVoro-D/AC/AI, here, as an example, we show how CCVD can be use to boost iVoro.

Original: In iVoro, the VD is induced by { c τ,k } τ ∈{ 1 ,...,t } ,k ∈{ 1 ,...,K τ } that are feature means from the last layer φ .
Revised: In iVoro, the VD is induced by { c τ,k } τ ∈{ 1 ,...,t } ,k ∈{ 1 ,...,K τ } that are feature means from the last layer ϕ .

Original: Now, we arbitrarily extract L layers { φ ( l ) } Ll =1 and generate the K totally ordered clusters {{ c ( l ) τ,k } Ll =1 } τ ∈{ 1 ,...,t } ,k ∈{ 1 ,...,K τ } to construct CCVD and generate the query cluster { φ ( l ) ( x ) } Ll =1 for the query example x for Voronoi cell assignment.
Revised: Now, we arbitrarily extract L layers { ϕ ( l ) } Ll =1 and generate the K totally ordered clusters {{ c ( l ) τ,k } Ll =1 } τ ∈{ 1 ,...,t } ,k ∈{ 1 ,...,K τ } to construct CCVD and generate the query cluster { ϕ ( l ) ( x ) } Ll =1 for the query example x for Voronoi cell assignment.

Original: See
Revised: See

Original: Appendix C for a summary of the notations and acronyms.
Revised: Appendix C for a summary of the notations and acronyms.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In our geometric framework, starting from iVoro, the simplest prototype-induced VD model, we gradually add four components: ( I ) parameterized normalization (iVoro-N), ( II ) divide-and-conquerfor progressive VD construction (iVoro-D), ( III ) augmentation consensus/integration (iVoro-AC/AI),
Revised: In our geometric framework, starting from iVoro, the simplest prototype-induced VD model, we gradually add four components: ( I ) parameterized normalization (iVoro-N), ( II ) divide-and-conquer

Original: [Table]
Revised: [Table]

Original:  and ( IV ) multilayer VD (iVoro-L).
Revised: for progressive VD construction (iVoro-D), ( III ) augmentation consensus/integration (iVoro-AC/AI), and ( IV ) multilayer VD (iVoro-L).

Original: In this section, our main goals are to: (1) validate the strength of every single component; (2) exhaust as many combinations of components as possible to see how different combinations collaboratively contribute to the overall result; and (3) investigate at which circumstances a method does or does not work, by analyzing data size, number of layers, and quantitative uncertainty.
Revised: In this section, our main goals are to: (1) validate the strength of every single component; (2) exhaust as many combinations of components as possible to see how different combinations collaboratively contribute to the overall result; and (3) investigate at which circumstances a method does or does not work, by analyzing data size, number of layers, and quantitative uncertainty.

Original: 3.1 Datasets, Benchmarks, and Implementation Details.
Revised: 3.1 Datasets, Benchmarks, and Implementation Details.

Original: Three standard datasets, CIFAR100 (Krizhevsky et al., 2009), TinyImageNet (Le & Yang, 2015) and ImageNet-Subset (Deng et al., 2009a) for CIL are used for method evaluation.
Revised: Three standard datasets, CIFAR100 (Krizhevsky et al., 2009), TinyImageNet (Le & Yang, 2015) and ImageNet-Subset (Deng et al., 2009a) for CIL are used for method evaluation.

Original: We follow the popular benchmarking protocol in exemplar-free CIL used by (Liu et al., 2021b; Zhu et al., 2021; Douillard et al., 2020; Hou et al., 2019) in which the inital phase contains a half of the classes while the subsequent phases each has 15 , 110 , or 120 of the remaining classes.
Revised: We follow the popular benchmarking protocol in exemplar-free CIL used by (Liu et al., 2021b; Zhu et al., 2021; Douillard et al., 2020; Hou et al., 2019) in which the inital phase contains a half of the classes while the subsequent phases each has 15 , 110 , or 120 of the remaining classes.

Original: We mainly compare our method to non-exemplar methods including EWC (Kirkpatrick et al., 2017), LwF (Li & Hoiem, 2017), LwF-MC (Li & Hoiem, 2017),
Revised: We mainly compare our method to non-exemplar methods including EWC (Kirkpatrick et al., 2017), LwF (Li & Hoiem, 2017), LwF-MC (Li & Hoiem, 2017),

Original: LwM (Dhar et al., 2019), and MUC (Liu et al., 2020b), but we also compare with several recent exemplar-based methods iCaRL (Rebufﬁ et al., 2017), EEIL (Castro et al., 2018), UCIR (Hou et al., 2019), and RMM (Liu et al., 2021b) for reference.
Revised: LwM (Dhar et al., 2019), and MUC (Liu et al., 2020b), but we also compare with several recent exemplar-based methods iCaRL (Rebuffi et al., 2017), EEIL (Castro et al., 2018), UCIR (Hou et al., 2019), and RMM (Liu et al., 2021b) for reference.

Original: A ResNet-18 (He et al., 2016) model is used for all experiments.
Revised: A ResNet-18 (He et al., 2016) model is used for all experiments.

Original: We follow PASS (Zhu et al., 2021) to train the feature extractor on the ﬁrst phase data but freeze it afterwards for all subsequent phases.
Revised: We follow PASS (Zhu et al., 2021) to train the feature extractor on the first phase data but freeze it afterwards for all subsequent phases.

Original: All classes are expanded via rotating the original image by 90°, 180°, and 270°.
Revised: All classes are expanded via rotating the original image by 90°, 180°, and 270°.

Original: See Appendix F for more details about the implementations of all the 12 ablation methods in Tab. 2.
Revised: See Appendix F for more details about the implementations of all the 12 ablation methods in Tab. 2.

Original: [Figure]
Revised: [Figure]

Original: 3.2 iVoro: Simple VD is A Strong Baseline.
Revised: 3.2 iVoro: Simple VD is A Strong Baseline.

Original: Surprisingly, by only using prototypes for VD construction, our baseline method iVoro can achieve competitive performance for short phases and much better results for long phases, compared to the state-of-the-art non-exemplar CIL method.
Revised: Surprisingly, by only using prototypes for VD construction, our baseline method iVoro can achieve competitive performance for short phases and much better results for long phases, compared to the state-of-the-art non-exemplar CIL method.

Original: For example, the difference in accuracy in comparison to PASS is 0.29%/6.91%/3.63% for 5/10/20-phase
Revised: For example, the difference in accuracy in comparison to PASS is 0.29%/6.91%/3.63% for 5/10/20-phase

Original: CIFAR-100, -3.58%/-1.09%/5.43% for 5/10/20-phase TinyImageNet, and 4.76% for ImageNetSubset.
Revised: CIFAR-100, -3.58%/-1.09%/5.43% for 5/10/20-phase TinyImageNet, and 4.76% for ImageNetSubset.

Original: We suspect that this is because the features generated by the frozen feature extractor can besatisfactorily separable by linear bisectors (Fig. 2). As we can see, the features for other methods are
Revised: We suspect that this is because the features generated by the frozen feature extractor can be

Original: [Table]
Revised: [Table]

Revised: satisfactorily separable by linear bisectors (Fig. 2).

Original:  all dramatically changing during the phases, but those for iVoro are all ﬁxed, making incremental VD construction possible.
Revised: As we can see, the features for other methods are all dramatically changing during the phases, but those for iVoro are all fixed, making incremental VD construction possible.

Original: Moreover, the accuracy of the last phase usually drops signiﬁcantly with longer task sequence (e.g. 20 phases vs. 5 phases), but iVoro is highly robust at the last phase, because the ﬁnal VDs are the same no matter how many phases it goes through.
Revised: Moreover, the accuracy of the last phase usually drops significantly with longer task sequence (e.g. 20 phases vs. 5 phases), but iVoro is highly robust at the last phase, because the final VDs are the same no matter how many phases it goes through.

Original: These results show that iVoro works favorably with long phases.
Revised: These results show that iVoro works favorably with long phases.

Original: When parameterized normalization is applied, iVoro-N further consistently improves upon iVoro by up to 2.40% (10-phase ImageNet-Subset) (see Tab. 2), by encouraging the compactness of feature distribution.
Revised: When parameterized normalization is applied, iVoro-N further consistently improves upon iVoro by up to 2.40% (10-phase ImageNet-Subset) (see Tab. 2), by encouraging the compactness of feature distribution.

Original: See Appendix H about the detailed analysis of iVoro-N.
Revised: See Appendix H about the detailed analysis of iVoro-N.

Original: [Figure]
Revised: [Figure]

Original: cy
Revised: r

Revised: [Figure]

Original: PASS i V o r o N D ND
Revised: i V o r o N D ND

Original: A C
Revised: A C

Original: A I
Revised: A I

Original: N A C
Revised: N A C

Original: N A I ND
Revised: N A I ND

Original: A
Revised: A

Original: I
Revised: I

Original: N A C L
Revised: N A C L

Original: N AI L ND A I L
Revised: N A

Original: CIFAR-100 (20 phases)
Revised: [Figure]

Original: iVoro
Revised: L

Original: [Figure]
Revised: [Figure]

Original: AC
Revised: A

Original: AI L
Revised: [Figure] L

Original: Methods
Revised: [Figure]

Original: A cc u r a cy

Original: PASS i V o r o N D ND
Revised: i V o r o N D ND

Original: A C
Revised: A C

Original: A I N A C
Revised: A I N A C

Original: N A I ND
Revised: N A I ND

Original: A
Revised: A

Original: I
Revised: I

Original: N A C L
Revised: N A C L

Original: N A I L ND
Revised: N A I L ND

Original: A I L
Revised: A I L

Original: [Figure] (20 phases)
Revised: TinyImageNet (20 phases)

Revised: iVoro

Original: [Figure]
Revised: [Figure]

Original:  AC
Revised: D AC

Original: [Figure] L
Revised: AI L

Original: [Figure]
Revised: Methods

Original: A cc u r a cy
Revised: A cc u r a cy

Original: PASS i V
Revised: PASS i V

Original: o r o N D ND
Revised: o r o N D ND

Original: A C
Revised: A C

Original: A I
Revised: A I

Original: N A C
Revised: N A C

Original: N A I ND
Revised: N A I ND

Original: A I N A C L
Revised: A I N A C L

Original: N A I L ND
Revised: N A I L ND

Original: A I L
Revised: A I L

Revised: ImageNet-Subset (10 phases)

Revised: iVoro

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: (iVoro-D): Synergistic Effects.
Revised: 3.3 Normalization (iVoro-N) and D&C (iVoroD): Synergistic Effects.

Original: Our very baseline method, iVoro, ignores at which phase a class was learned, and computes prototypes indifferently to construct the VD (i.e. 1-nearest neighbor model).
Revised: Our very baseline method, iVoro, ignores at which phase a class was learned, and computes prototypes indifferently to construct the VD (i.e. 1-nearest neighbor model).

Original: To determine the decision boundaries more subtly, iVoro-D focuses on the reﬁnement of the within-phase boundaries.
Revised: To determine the decision boundaries more subtly, iVoro-D focuses on the refinement of the within-phase boundaries.

Original: These two components, iVoro-N/D, can individually improve iVoro, but also have collective impacts.
Revised: These two components, iVoro-N/D, can individually improve iVoro, but also have collective impacts.

Original: For example, as shown in Tab. 2 and Fig. 6, iVoro-ND > iVoro-N/iVoro-D > iVoro, corroborating that every single contribution is useful and necessary.
Revised: For example, as shown in Tab. 2 and Fig. 6, iVoro-ND > iVoro-N/iVoro-D > iVoro, corroborating that every single contribution is useful and necessary.

Original: More speciﬁcally, in the three datasets, iVoro-D makes the largest contribution to TinyImageNet (1.75%-3.44%) than to CIFAR-100 (0.48%-0.96%) or to ImageNet-Subset (0.72%).
Revised: More specifically, in the three datasets, iVoro-D makes the largest contribution to TinyImageNet (1.75%-3.44%) than to CIFAR-100 (0.48%-0.96%) or to ImageNet-Subset (0.72%).

Original: This can be explained by the fact that there are 100 classes in the ﬁrst phase in the TinyImageNet dataset, while only 50 in the other two, making TinyImageNet a harder dataset if only vanilla prototypes are used to construct the VD.
Revised: This can be explained by the fact that there are 100 classes in the first phase in the TinyImageNet dataset, while only 50 in the other two, making TinyImageNet a harder dataset if only vanilla prototypes are used to construct the VD.

Original: 4 Why and When Will Augmentation Integration (iVoro AC/AI) Help?
Revised: 3.4 Why and When Will Augmentation Integration (iVoro-AC/AI) Help?

Original: When augmentation consensus (iVoro-AC) or integration (iVoro-AI) is applied, the improvement is signiﬁcant.
Revised: When augmentation consensus (iVoro-AC) or integration (iVoro-AI) is applied, the improvement is significant.

Original: For example, iVoro-AC obtains 13.76%, 17.00%, and 16.50% improvements upon iVoro on CIFAR-100,
Revised: For example, iVoro-AC obtains 13.76%, 17.00%, and 16.50% improvements upon iVoro on CIFAR-100,

Original: TinyImageNet, and ImageNet-Subset, respectively.
Revised: TinyImageNet, and ImageNet-Subset, respectively.

Original: iVoro-AI itself is worse than iVoro-AC, but if combined with normalization and D&C, it further elevates the accuracy by a large margin, e.g. as high as 68.70% (iVoro-NDAI) on 20-phase TinyImageNet and 78.64% on 10-phase ImageNet-Subset.
Revised: iVoro-AI itself is worse than iVoro-AC, but if combined with normalization and D&C, it further elevates the accuracy by a large margin, e.g. as high as 68.70% (iVoro-NDAI) on 20-phase TinyImageNet and 78.64% on 10-phase ImageNet-Subset.

Original: To investigate the reason of this prominent improvement, we calculate the entropy-based geometric variance in class level and plot them as a function of the ∆ accuracy (i.e. the improvement in accuracy after augmentation integration is used), as shown in Fig.
Revised: To investigate the reason of this prominent improvement, we calculate the entropy-based geometric variance in class level and plot them as a function of the ∆ accuracy (i.e. the improvement in accuracy after augmentation integration is used), as shown in Fig.

Original: 8 and Fig.
Revised: 8 and Fig.

Original: G.2.
Revised: G.2.

Original: Interestingly, there is a clear correlation between HV and ∆ accuracy, and this is more notable on ImageNet-Subset (Pearson’s R ∼ 0 . 9 ), probably because of its high resolution ( 224 × 224 ).
Revised: Interestingly, there is a clear correlation between HV and ∆ accuracy, and this is more notable on ImageNet-Subset (Pearson’s R ∼ 0 . 9 ), probably because of its high resolution ( 224 × 224 ).

Original: This tendency suggests that the higher the variance within the assignments from augmented images to expanded classes, the better the improvement after using augmentation integration.
Revised: This tendency suggests that the higher the variance within the assignments from augmented images to expanded classes, the better the improvement after using augmentation integration.

Original: See Appendix I for uncertainty analysis, and
Revised: See Appendix I for uncertainty analysis, and

Original: Appendix N/Appendix O for class-level/sample-level analysis.
Revised: Appendix N/Appendix O for class-level/sample-level analysis.

Original: [Figure]
Revised: [Figure]

Original: 3.5 How Good Should the Feature Extractor Be?
Revised: 3.5 How Good Should the Feature Extractor Be?

Original: As iVoro is heavily dependent on the feature extractor, which cannot be evolved in any way along the learning process, one may wonder if our method still work with a poorly trained feature extractor.
Revised: As iVoro is heavily dependent on the feature extractor, which cannot be evolved in any way along the learning process, one may wonder if our method still work with a poorly trained feature extractor.

Original: To verify this, we gradually decrease the number of classes used to train the feature extractor φ .
Revised: To verify this, we gradually decrease the number of classes used to train the feature extractor ϕ .

Original: As shown in Appendix J, compared with PASS, the best version of iVoro still has 17.75%, 13.59%, 10.89%, and 1.60% improvements with 40, 30, 20, and 10 initial classes, respectively.
Revised: As shown in Appendix J, compared with PASS, the best version of iVoro still has 17.75%, 13.59%, 10.89%, and 1.60% improvements with 40, 30, 20, and 10 initial classes, respectively.

Original: This means that, even if there is no strong feature extractor, our method can still reach acceptable performance higher than the state-of-the-art method.
Revised: This means that, even if there is no strong feature extractor, our method can still reach acceptable performance higher than the state-of-the-art method.

Original: [Table]
Revised: [Table]

Revised: 3.6 iVoro-L: VD Can Also

Revised: Go Deeper.

Original: In this section, we extract the feature from the 3 rd block and establish a Cluster-to-cluster
Revised: In this section, we extract the feature from the 3 rd block and establish a Cluster-to-cluster

Original: Voronoi Diagram (CCVD) using features from both layers.
Revised: Voronoi Diagram (CCVD) using features from both layers.

Original: As expected, shown in Appendix K, the overall performance degenerates substantially, e.g. iVoro-NAI drops from 65.84% to 42.03%.
Revised: As expected, shown in Appendix K, the overall performance degenerates substantially, e.g. iVoro-NAI drops from 65.84% to 42.03%.

Original: However, and interestingly, if integrated with CCVD and D&C, iVoro-NDAIL obtains even higher accuracy of 72.34%.
Revised: However, and interestingly, if integrated with CCVD and D&C, iVoro-NDAIL obtains even higher accuracy of 72.34%.

Original: The ﬁnal results are presented in Tab. 1, Fig.
Revised: The final results are presented in Tab. 1, Fig.

Original: K.3, and Appendix G. Our ﬁnal model, multilayer VD, surpasses all previous methods by a large margin of 25 .
Revised: K.3, and Appendix G. Our final model, multilayer VD, surpasses all previous methods by a large margin of 25 .

Original: 26% , 37 . 09% , and 33 .
Revised: 26% , 37 . 09% , and 33 .

Original: 21% on CIFAR-100,
Revised: 21% on CIFAR-100,

Original: TinyImageNet, and ImageNet-Subset, respectively, even higher than exemplar-based CIL methods.
Revised: TinyImageNet, and ImageNet-Subset, respectively, even higher than exemplar-based CIL methods.

Original: 3.7 Comparison with Joint Training.
Revised: 3.7 Comparison with Joint Training.

Original: In CIL, the classes are sequentially learned at each phase, whereas joint training simultaneously learns all the classes in the same phase, providing an upper bound for our CIL experiments.
Revised: In CIL, the classes are sequentially learned at each phase, whereas joint training simultaneously learns all the classes in the same phase, providing an upper bound for our CIL experiments.

Original: In Tab. 3, iVoro (and its variants) is applied to joint training, and is compared with ResNet on both the original and expanded label sets.
Revised: In Tab. 3, iVoro (and its variants) is applied to joint training, and is compared with ResNet on both the original and expanded label sets.

Original: Although there is still a substantial gap between iVoro (best) and the upper bound, the catastrophic forgetting is considerably overcome (see Appendix M).
Revised: Although there is still a substantial gap between iVoro (best) and the upper bound, the catastrophic forgetting is considerably overcome (see Appendix M).

Original: In addition, and surprisingly, iVoro-AC/AI can also promote the performance of joint training, e.g. +6 .
Revised: In addition, and surprisingly, iVoro-AC/AI can also promote the performance of joint training, e.g. +6 .

Original: 31% , +17 . 54% , and +13 .
Revised: 31% , +17 . 54% , and +13 .

Original: 38% for CIFAR-100, TinyImageNet, and ImageNet-Subset, respectively, suggesting that our augmentation integration method is also beneﬁcial to general training where self-supervised label augmentation is involved.
Revised: 38% for CIFAR-100, TinyImageNet, and ImageNet-Subset, respectively, suggesting that our augmentation integration method is also beneficial to general training where self-supervised label augmentation is involved.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we use progressive Voronoi Diagram to model the class-incremental learning problem, and propose a number of new techniques that handle various aspects of this VD construction process that gradually and greatly improve the CIL performance.
Revised: In this paper, we use progressive Voronoi Diagram to model the class-incremental learning problem, and propose a number of new techniques that handle various aspects of this VD construction process that gradually and greatly improve the CIL performance.

Original: Thus, iVoro is shown to be a ﬂexible, scalable, and robust framework that strictly maintains the privacy of previous data.
Revised: Thus, iVoro is shown to be a flexible, scalable, and robust framework that strictly maintains the privacy of previous data.

Original: Our code is available at GitHub.
Revised: Our code is available at https://machunwei.github.io/ivoro/.

Revised: A CKNOWLEDGMENTS

Revised: This research was supported in part by NSF through grant IIS-1910492.

Original: A E XTENDED R ELATED W ORK A. R ECENT P ROGRESS IN I NCREMENTAL L EARNING
Revised: A E XTENDED R ELATED W ORK A. R ECENT P ROGRESS IN I NCREMENTAL L EARNING

Original: Incremental Learning (Rebufﬁ et al., 2017; Hou et al., 2019; Wu et al., 2019; Zhu et al., 2021; Liu et al., 2021b) requires continuously updating a model using a sequence of new tasks without forgetting the old knowledge, which is also referred to as continual learning (Parisi et al., 2019; Delange et al., 2021; Chaudhry et al., 2019).
Revised: Incremental Learning (Rebuffi et al., 2017; Hou et al., 2019; Wu et al., 2019; Zhu et al., 2021; Liu et al., 2021b) requires continuously updating a model using a sequence of new tasks without forgetting the old knowledge, which is also referred to as continual learning (Parisi et al., 2019; Delange et al., 2021; Chaudhry et al., 2019).

Original: The main challenge of incremental learning is catastrophic forgetting (McCloskey & Cohen, 1989; French, 1999; Goodfellow et al., 2014; Kemker et al., 2018), where deep neural network is prone to performance deterioration on the previously learned tasks as the model parameters overﬁt to the current data to optimize the stability-plasticity trade-off.
Revised: The main challenge of incremental learning is catastrophic forgetting (McCloskey & Cohen, 1989; French, 1999; Goodfellow et al., 2014; Kemker et al., 2018), where deep neural network is prone to performance deterioration on the previously learned tasks as the model parameters overfit to the current data to optimize the stability-plasticity trade-off.

Original: A.1. C AUSATION OF C ATASTROPHIC F ORGETTING
Revised: A.1. C AUSATION OF C ATASTROPHIC F ORGETTING

Original: Generally speaking, in deep neural networks, catastrophic forgetting (McCloskey & Cohen, 1989;
Revised: Generally speaking, in deep neural networks, catastrophic forgetting (McCloskey & Cohen, 1989;

Original: Goodfellow et al., 2014; Kemker et al., 2018) comes from two sources: the feature distribution shifting of the old classes in the feature embedding space as well as the confusion and imbalance of the decision boundary of the classiﬁer when learning new task.
Revised: Goodfellow et al., 2014; Kemker et al., 2018) comes from two sources: the feature distribution shifting of the old classes in the feature embedding space as well as the confusion and imbalance of the decision boundary of the classifier when learning new task.

Original: The former is caused by the excessive plasticity and parameter changing of the feature extractor of the deep model during ﬁnetuning on unseen data/classes, thus deteriorates the feature extraction and prediction on previous classes; while the latter is due to the highly overﬁtting and bias of the classiﬁer on current task as well as the overlapping between the representation of new and old classes in the feature space.
Revised: The former is caused by the excessive plasticity and parameter changing of the feature extractor of the deep model during finetuning on unseen data/classes, thus deteriorates the feature extraction and prediction on previous classes; while the latter is due to the highly overfitting and bias of the classifier on current task as well as the overlapping between the representation of new and old classes in the feature space.

Original: A.1. I NCREMENTAL L EARNING S CENARIOS
Revised: A.1. I NCREMENTAL L EARNING S CENARIOS

Original: Three common incremental learning scenarios are widely explored in recent papers (Van de Ven &
Revised: Three common incremental learning scenarios are widely explored in recent papers (Van de Ven &

Original: Tolias, 2019).
Revised: Tolias, 2019).

Original: Task-incremental learning (TIL) (Ostapenko et al., 2019; Shin et al., 2017; Kirkpatrick et al., 2017;
Revised: Task-incremental learning (TIL) (Ostapenko et al., 2019; Shin et al., 2017; Kirkpatrick et al., 2017;

Original: Zenke et al., 2017; Wu et al., 2018; Lopez-Paz & Ranzato, 2017; Chaudhry et al., 2019; Buzzega et al., 2020; Cha et al., 2021; Pham et al., 2021; Fernando et al., 2017) incrementally learns a sequence of tasks in multiple phases, where each task contains unseen data of a new set of classes.
Revised: Zenke et al., 2017; Wu et al., 2018; Lopez-Paz & Ranzato, 2017; Chaudhry et al., 2019; Buzzega et al., 2020; Cha et al., 2021; Pham et al., 2021; Fernando et al., 2017) incrementally learns a sequence of tasks in multiple phases, where each task contains unseen data of a new set of classes.

Original: To mitigate catastrophic forgetting, TIL assumes a simple setting where the task identity is known at inference time.
Revised: To mitigate catastrophic forgetting, TIL assumes a simple setting where the task identity is known at inference time.

Original: The methods under this scenario keep leaning new task-independent classiﬁers or growing the model capacity by attaching additional modules (e.g. kernels, layers or branches), each corresponding to a speciﬁc task or a subset of classes.
Revised: The methods under this scenario keep leaning new task-independent classifiers or growing the model capacity by attaching additional modules (e.g. kernels, layers or branches), each corresponding to a specific task or a subset of classes.

Original: Since the task ID is available during inference, the model can directly select proper classiﬁer or module without inferring task identity, which effectively solves the confusion boundary and classiﬁer bias between old and new tasks, and often achieves satisfying performance.
Revised: Since the task ID is available during inference, the model can directly select proper classifier or module without inferring task identity, which effectively solves the confusion boundary and classifier bias between old and new tasks, and often achieves satisfying performance.

Original: However, knowing task identity at test time is normally unrealistic in real-world situation hence restricts practical usage.
Revised: However, knowing task identity at test time is normally unrealistic in real-world situation hence restricts practical usage.

Original: Moreover, it may incur unbounded memory consumption for super long task sequence if increasing the model capacity for new tasks.
Revised: Moreover, it may incur unbounded memory consumption for super long task sequence if increasing the model capacity for new tasks.

Original: Unlike TIL constrained by the availability of task identity, class-incremental learning (CIL) (Liu et al., 2020b; Belouadah & Popescu, 2019; Chaudhry et al., 2018a; Zhu et al., 2021; Douillard et al., 2020; Rebufﬁ et al., 2017; Hou et al., 2019; Liu et al., 2020a; 2021a;b) updates a uniﬁed classiﬁer for all classes learned so far while task identity is no longer required during inference.
Revised: Unlike TIL constrained by the availability of task identity, class-incremental learning (CIL) (Liu et al., 2020b; Belouadah & Popescu, 2019; Chaudhry et al., 2018a; Zhu et al., 2021; Douillard et al., 2020; Rebuffi et al., 2017; Hou et al., 2019; Liu et al., 2020a; 2021a;b) updates a unified classifier for all classes learned so far while task identity is no longer required during inference.

Original: To compensate the missing task identity and alleviate forgetting issue, a branch of works (Rebufﬁ et al., 2017; Hou et al., 2019; Liu et al., 2021b; Douillard et al., 2020; Castro et al., 2018; Wu et al., 2019) alternatively follow a memory-based setting, in which a limited number of samples from old classes (e.g., 20 exemplars per class) is stored and maintained in a memory buffer, which are later replayed to jointly train the model with current data (normally combined with knowledge distillation) in order to constrain the feature distribution shifting of the old classes and the decision boundary bias of the classiﬁer.
Revised: To compensate the missing task identity and alleviate forgetting issue, a branch of works (Rebuffi et al., 2017; Hou et al., 2019; Liu et al., 2021b; Douillard et al., 2020; Castro et al., 2018; Wu et al., 2019) alternatively follow a memory-based setting, in which a limited number of samples from old classes (e.g., 20 exemplars per class) is stored and maintained in a memory buffer, which are later replayed to jointly train the model with current data (normally combined with knowledge distillation) in order to constrain the feature distribution shifting of the old classes and the decision boundary bias of the classifier.

Original: However, their performance deteriorates with smaller buffer size, and eventually, the storing and sharing of previous data, e.g. medical images, may not be feasible when memory limits and privacy issue are taken into consideration.
Revised: However, their performance deteriorates with smaller buffer size, and eventually, the storing and sharing of previous data, e.g. medical images, may not be feasible when memory limits and privacy issue are taken into consideration.

Original: Given the potential memory issue, another direction of works (Kirkpatrick et al., 2017; Zenke et al., 2017; Li & Hoiem, 2017; Dhar et al., 2019; Zhu et al., 2021) intend to explore CIL in a much challenging setting without memory rehearsal, mainly based on regularization and knowledge distillation techniques, which is known as exemplar-free CIL.
Revised: Given the potential memory issue, another direction of works (Kirkpatrick et al., 2017; Zenke et al., 2017; Li & Hoiem, 2017; Dhar et al., 2019; Zhu et al., 2021) intend to explore CIL in a much challenging setting without memory rehearsal, mainly based on regularization and knowledge distillation techniques, which is known as exemplar-free CIL.

Original: In this paper, we are following this CIL setting.
Revised: In this paper, we are following this CIL setting.

Original: Domain-incremental learning (DIL) (Rostami, 2021; Tang et al., 2021; Volpi et al., 2021), different from the aforementioned two scenarios, incrementally learning new domains of the same classes in each phase.
Revised: Domain-incremental learning (DIL) (Rostami, 2021; Tang et al., 2021; Volpi et al., 2021), different from the aforementioned two scenarios, incrementally learning new domains of the same classes in each phase.

Original: Some domain adaptation techniques, e.g. meta learning, data shifting, domain randomization, are implemented in DIL to increase the model robustness and generalizability to handle various domain distributions.
Revised: Some domain adaptation techniques, e.g. meta learning, data shifting, domain randomization, are implemented in DIL to increase the model robustness and generalizability to handle various domain distributions.

Original: Since this scenario is not quite related to this paper, no detailed discussion will be included.
Revised: Since this scenario is not quite related to this paper, no detailed discussion will be included.

Original: A.1. C ATEGORIES OF I NCREMENTAL L EARNING M ETHODS
Revised: A.1. C ATEGORIES OF I NCREMENTAL L EARNING M ETHODS

Original: There are three categories of existing IL methods to overcome catastrophic forgetting (Delange et al., 2021).
Revised: There are three categories of existing IL methods to overcome catastrophic forgetting (Delange et al., 2021).

Original: Regularization-based methods constrain the plasticity of the model to preserve old knowledge.
Revised: Regularization-based methods constrain the plasticity of the model to preserve old knowledge.

Original: This can be addressed by directly penalizing the changes of important parameters for previous tasks (Aljundi et al., 2018; Chaudhry et al., 2018a;
Revised: This can be addressed by directly penalizing the changes of important parameters for previous tasks (Aljundi et al., 2018; Chaudhry et al., 2018a;

Original: Kirkpatrick et al., 2017; Zenke et al., 2017;
Revised: Kirkpatrick et al., 2017; Zenke et al., 2017;

Original: Kumar et al., 2021) or regularizing the gradients when training on unseen data (Lopez-Paz & Ranzato, 2017; Chaudhry et al., 2018b).
Revised: Kumar et al., 2021) or regularizing the gradients when training on unseen data (Lopez-Paz & Ranzato, 2017; Chaudhry et al., 2018b).

Original: Knowledge distillation is another regularization solution, which is widely used in various IL methods to implicitly consolidate previous knowledge by introducing regularization loss term on model representations, including output logits or probabilities (Li &
Revised: Knowledge distillation is another regularization solution, which is widely used in various IL methods to implicitly consolidate previous knowledge by introducing regularization loss term on model representations, including output logits or probabilities (Li &

Original: Hoiem, 2017; Schwarz et al., 2018; Rebufﬁ et al., 2017; Castro et al., 2018) and intermediate features (Hou et al., 2019; Dhar et al., 2019; Douillard et al., 2020; Zhu et al., 2021).
Revised: Hoiem, 2017; Schwarz et al., 2018; Rebuffi et al., 2017; Castro et al., 2018) and intermediate features (Hou et al., 2019; Dhar et al., 2019; Douillard et al., 2020; Zhu et al., 2021).

Original: Some other works focus on correcting the classiﬁer bias on new classes (Belouadah & Popescu, 2019;
Revised: Some other works focus on correcting the classifier bias on new classes (Belouadah & Popescu, 2019;

Original: Wu et al., 2019; Belouadah & Popescu, 2020;
Revised: Wu et al., 2019; Belouadah & Popescu, 2020;

Original: Zhao et al., 2020).
Revised: Zhao et al., 2020).

Original: Rehearsal-based methods either store and replay a limited amount of exemplars from old classes as raw images (Rebufﬁ et al., 2017;
Revised: Rehearsal-based methods either store and replay a limited amount of exemplars from old classes as raw images (Rebuffi et al., 2017;

Original: Hou et al., 2019; Liu et al., 2021b; Chaudhry et al., 2019; Buzzega et al., 2020) or embedded features (Hayes et al., 2020; Iscen et al., 2020) to jointly train the model in the incremental phases, or alternatively generate exemplars of previous classes (Ostapenko et al., 2019; Shin et al., 2017; Wu et al., 2018; Kemker & Kanan, 2017).
Revised: Hou et al., 2019; Liu et al., 2021b; Chaudhry et al., 2019; Buzzega et al., 2020) or embedded features (Hayes et al., 2020; Iscen et al., 2020) to jointly train the model in the incremental phases, or alternatively generate exemplars of previous classes (Ostapenko et al., 2019; Shin et al., 2017; Wu et al., 2018; Kemker & Kanan, 2017).

Original: The former relies on memory buffer for all learned classes, where the performance is constrained by the buffer size limits and it is impracticable when data privacy is required and storing data is prohibited.
Revised: The former relies on memory buffer for all learned classes, where the performance is constrained by the buffer size limits and it is impracticable when data privacy is required and storing data is prohibited.

Original: The latter requires continuously learning a deep generative model, which is also prone to catastrophic forgetting thus the quality of generated exemplars is not reliable.
Revised: The latter requires continuously learning a deep generative model, which is also prone to catastrophic forgetting thus the quality of generated exemplars is not reliable.

Original: Architecture-based methods aims at dynamically adapting task-speciﬁc sub-network architectures, which requires task identity to select proper sub-network.
Revised: Architecture-based methods aims at dynamically adapting task-specific sub-network architectures, which requires task identity to select proper sub-network.

Original: Some works directly expand the network by adding new layers or branches (Rusu et al., 2016;
Revised: Some works directly expand the network by adding new layers or branches (Rusu et al., 2016;

Original: Li et al., 2019; Wang et al., 2017; Yoon et al., 2017), which is limited in practice due to unbounded model parameter growth.
Revised: Li et al., 2019; Wang et al., 2017; Yoon et al., 2017), which is limited in practice due to unbounded model parameter growth.

Original: Others freeze partial network with masks for old tasks (Golkar et al., 2019; Hung et al., 2019; Mallya & Lazebnik, 2018;
Revised: Others freeze partial network with masks for old tasks (Golkar et al., 2019; Hung et al., 2019; Mallya & Lazebnik, 2018;

Original: Serra et al., 2018), but suffering from running out of model parameters for new knowledge.
Revised: Serra et al., 2018), but suffering from running out of model parameters for new knowledge.

Original: The architecture-based methods are usually combined with memory buffer and distillation, and can achieve good results.
Revised: The architecture-based methods are usually combined with memory buffer and distillation, and can achieve good results.

Original: Our work is focusing on the most challenging but also the most practical non-exemplar classincremental learning problem, which is a general real-world scenario when no old data can be stored due to memory limits or data privacy and task identity is unavailable during inference, with the constraint of ﬁxed model capacity in the same time.
Revised: Our work is focusing on the most challenging but also the most practical non-exemplar classincremental learning problem, which is a general real-world scenario when no old data can be stored due to memory limits or data privacy and task identity is unavailable during inference, with the constraint of fixed model capacity in the same time.

Original: A. C OMPUTATIONAL G EOMETRY FOR D EEP L EARNING
Revised: A. C OMPUTATIONAL G EOMETRY FOR D EEP L EARNING

Original: Computational geometry is an emerging perspective for studying various aspects of deep learning.
Revised: Computational geometry is an emerging perspective for studying various aspects of deep learning.

Original: The geometric structure of deep neural networks is ﬁrst hinted at by (Raghu et al., 2017) which reveals that piecewise linear activations subdivide input space into convex polytopes.
Revised: The geometric structure of deep neural networks is first hinted at by (Raghu et al., 2017) which reveals that piecewise linear activations subdivide input space into convex polytopes.

Original: Afterward, (Balestriero et al., 2019) points out that the exact structure is a Power Diagram (PD, a generalized form of
Revised: Afterward, (Balestriero et al., 2019) points out that the exact structure is a Power Diagram (PD, a generalized form of

Original: Voronoi Diagram)
Revised: Voronoi Diagram)

Original: (Aurenhammer, 1987) which is subsequently used to explain the recurrent neural networks (Wang et al., 2018) and generative models (Balestriero et al., 2020).
Revised: (Aurenhammer, 1987) which is subsequently used to explain the recurrent neural networks (Wang et al., 2018) and generative models (Balestriero et al., 2020).

Original: The Power Diagram (or Voronoi Diagram) subdivision, however, is not necessarily the optimal model for describing the partitioning of deep/intermediate feature spaces.
Revised: The Power Diagram (or Voronoi Diagram) subdivision, however, is not necessarily the optimal model for describing the partitioning of deep/intermediate feature spaces.

Original: More recently, several works in computational geometry (Chen et al., 2013; 2017; Huang et al., 2021) use an inﬂuence function F ( C , z ) to measure the joint inﬂuence of all objects in C on a query z to build a Cluster-induced Voronoi Diagram (CIVD), providing an advanced reform of the classical Voronoi Diagram.
Revised: More recently, several works in computational geometry (Chen et al., 2013; 2017; Huang et al., 2021) use an influence function F ( C , z ) to measure the joint influence of all objects in C on a query z to build a Cluster-induced Voronoi Diagram (CIVD), providing an advanced reform of the classical Voronoi Diagram.

Original: Observing that Prototypical
Revised: Observing that Prototypical

Original: Network (Snell et al., 2017), a widely adopted metric-based few-shot learning (FSL) method, is essentially a Voronoi Diagram in the feature space, DeepVoro (Ma et al., 2022b) ﬁrst uniﬁes various kinds of FSL methods, and then constructs a CIVD by incorporating heterogeneous features, achieving the state-of-the-art performance in FSL.
Revised: Network (Snell et al., 2017), a widely adopted metric-based few-shot learning (FSL) method, is essentially a Voronoi Diagram in the feature space, DeepVoro (Ma et al., 2022b) first unifies various kinds of FSL methods, and then constructs a CIVD by incorporating heterogeneous features, achieving the state-of-the-art performance in FSL.

Original: Besides FSL, Voronoi Diagram subdivision has also been used for deep learning uncertainty calibration (Ma et al., 2021), adversarial robustness (Sitawarin et al., 2021), topological data analysis (Polianskii & Pokorny, 2019; 2020; Poklukar et al., 2022), and medical applications (Ma et al., 2018; 2019).
Revised: Besides FSL, Voronoi Diagram subdivision has also been used for deep learning uncertainty calibration (Ma et al., 2021), adversarial robustness (Sitawarin et al., 2021), topological data analysis (Polianskii & Pokorny, 2019; 2020; Poklukar et al., 2022), and medical applications (Ma et al., 2018; 2019).

Original: In this paper, distinct from the three aforementioned lines of research (i.e. regularization-based, rehearsal-based, and architecture-based methods), we propose the geometry-based CIL method iVoro (and its variants), inspired by Voronoi Diagram subdivision.
Revised: In this paper, distinct from the three aforementioned lines of research (i.e. regularization-based, rehearsal-based, and architecture-based methods), we propose the geometry-based CIL method iVoro (and its variants), inspired by Voronoi Diagram subdivision.

Original: B D EMONSTRATIVE I LLUSTRATION ON MNIST D ATASET IN 2D S PACE
Revised: B D EMONSTRATIVE I LLUSTRATION ON MNIST D ATASET IN 2D S PACE

Original: In Figure2, MNIST (LeCun, 1998), a small and simple dataset, was used for the illustration of four methods, ﬁne-tuning, PASS (Zhu et al., 2021), iVoro, and iVoro-AC, because of the convenience of embedding the examples into R 2 .
Revised: In Figure2, MNIST (LeCun, 1998), a small and simple dataset, was used for the illustration of four methods, fine-tuning, PASS (Zhu et al., 2021), iVoro, and iVoro-AC, because of the convenience of embedding the examples into R 2 .

Original: The total 10 classes are split into a sequence of 4, 3, and 3 classes.
Revised: The total 10 classes are split into a sequence of 4, 3, and 3 classes.

Original: A ResNet-18 model is used as the feature extractor for all four methods.
Revised: A ResNet-18 model is used as the feature extractor for all four methods.

Original: ( A )
Revised: ( A )

Original: In ﬁne-tuning, the model is ﬁrstly trained on the 4 classes in the ﬁrst phase, and then ﬁne-tuned only on the subsequent 3 and 3 classes in phase 2 and phase 3.
Revised: In fine-tuning, the model is firstly trained on the 4 classes in the first phase, and then fine-tuned only on the subsequent 3 and 3 classes in phase 2 and phase 3.

Original: ( B ) In PASS, SSL-based label augmentation is applied on all three phases and expands the classes to be 16, 9, and 9 classes.
Revised: ( B ) In PASS, SSL-based label augmentation is applied on all three phases and expands the classes to be 16, 9, and 9 classes.

Original: The default hyper-parameters are used to train PASS (i.e. the weight for knowledge distillation is 10 and the weight for prototype augmentation is 10).
Revised: The default hyper-parameters are used to train PASS (i.e. the weight for knowledge distillation is 10 and the weight for prototype augmentation is 10).

Original: To ensure the ﬁnal subdivision of space is a Voronoi diagram, Thm. 3.1 (i.e.
Revised: To ensure the final subdivision of space is a Voronoi diagram, Thm. 2.1 (i.e.

Original: Voronoi diagram reduction in Algorithm 1) is applied during the training of ﬁne-tuning and PASS.
Revised: Voronoi diagram reduction in Algorithm 1) is applied during the training of fine-tuning and PASS.

Original: ( C )
Revised: ( C )

Original: In iVoro, the feature extractor from the ﬁrst phase of (B) is frozen and used without ﬁne-tuning for all the subsequent phases.
Revised: In iVoro, the feature extractor from the first phase of (B) is frozen and used without fine-tuning for all the subsequent phases.

Original: The feature means are calculated as prototypes and no feature transformation is used.
Revised: The feature means are calculated as prototypes and no feature transformation is used.

Original: Note that only the features from the original images without rotation are used in iVoro.
Revised: Note that only the features from the original images without rotation are used in iVoro.

Original: ( D )
Revised: ( D )

Original: The only difference with (C) is that all the expanded classes are also considered as independent cells, allowing for further integration.
Revised: The only difference with (C) is that all the expanded classes are also considered as independent cells, allowing for further integration.

Original: Result Analysis.
Revised: Result Analysis.

Original: For (A) ﬁne-tuning and (B) PASS, the model’s accuracy for data at individual phases are also shown in shadow.
Revised: For (A) fine-tuning and (B) PASS, the model’s accuracy for data at individual phases are also shown in shadow.

Original: Fine-tuning are able to achieve near-perfect prediction for classes in the current phase locally, but fails to maintain satisfactory performance on any historical class (accuracy ∼ 0% ).
Revised: Fine-tuning are able to achieve near-perfect prediction for classes in the current phase locally, but fails to maintain satisfactory performance on any historical class (accuracy ∼ 0% ).

Original: PASS, on the other hand, basically deteriorates slightly on the classes from the ﬁrst phase, due to the high weights on the KD loss and the prototype loss, but it also becomes almost incapable of learning on new classes (accuracy ∼ 0% ).
Revised: PASS, on the other hand, basically deteriorates slightly on the classes from the first phase, due to the high weights on the KD loss and the prototype loss, but it also becomes almost incapable of learning on new classes (accuracy ∼ 0% ).

Original: iVoro, i.e. the simplest 1-nearest-neighbor model, surprisingly obtains superior accuracy (64.84%, 24.28% higher than PASS) by only using a ﬁxed feature extractor trained from only 4 classes (16 expanded classes).
Revised: iVoro, i.e. the simplest 1-nearest-neighbor model, surprisingly obtains superior accuracy (64.84%, 24.28% higher than PASS) by only using a fixed feature extractor trained from only 4 classes (16 expanded classes).

Original: iVoro-AC achieves comparable result (61.77%) with iVoro, but the 2D embedding makes it harder to demonstrate the efﬁcacy of our proposed method.
Revised: iVoro-AC achieves comparable result (61.77%) with iVoro, but the 2D embedding makes it harder to demonstrate the efficacy of our proposed method.

Original: From this 2D illustration, it is obvious that the much higher performance of iVoro/iVoro-AC is achieved through a much better space partitioning.
Revised: From this 2D illustration, it is obvious that the much higher performance of iVoro/iVoro-AC is achieved through a much better space partitioning.

Original: C N OTATIONS AND A CRONYMS
Revised: C N OTATIONS AND A CRONYMS

Original: In this section, we list all the notations used in the Methodology in Tab.
Revised: In this section, we list all the notations used in the Methodology in Tab.

Original: C.1, the notations and acronyms for various geometric structures used in the paper in Tab.
Revised: C.1, the notations and acronyms for various geometric structures used in the paper in Tab.

Original: C.2, and all ablation methods in
Revised: C.2, and all ablation methods in

Original: Tab.
Revised: Tab.

Original: C.3.
Revised: C.3.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: D D ATASET D ETAILS
Revised: D D ATASET D ETAILS

Original: Here we give the detailed statistics of the three datasets used in the paper.
Revised: Here we give the detailed statistics of the three datasets used in the paper.

Original: Augmentation consensus (iVoro-AC) and augmentation integration (iVoro-AI) work more favorably with images with higher resolution, e.g. ImageNet-Subset (see Fig.
Revised: Augmentation consensus (iVoro-AC) and augmentation integration (iVoro-AI) work more favorably with images with higher resolution, e.g. ImageNet-Subset (see Fig.

Original: 8), as the rotation operation makes less sense if the image is too blur.
Revised: 8), as the rotation operation makes less sense if the image is too blur.

Original: [Table]
Revised: [Table]

Original: E P OWER D IAGRAM S UBDIVISION AND V ORONOI R EDUCTION
Revised: E P OWER D IAGRAM S UBDIVISION AND V ORONOI R EDUCTION

Original: In this section we provide the proof of Theorem 3.1.
Revised: In this section we provide the proof of Theorem 2.1.

Original: Lemma E.1.
Revised: Lemma E.1.

Original: The vertical projection from the lower envelope of the hyperplanes { Π k ( z ) : W Tk z + b k } Kk =1 onto the input space R n deﬁnes the cells of a PD.
Revised: The vertical projection from the lower envelope of the hyperplanes { Π k ( z ) : W Tk z + b k } Kk =1 onto the input space R n defines the cells of a PD.

Original: Theorem 2.1 (Voronoi Diagram Reduction (Ma et al., 2022a)).
Revised: Theorem 2.1 (Voronoi Diagram Reduction (Ma et al., 2022a)).

Original: The linear classiﬁer parameterized by W , b partitions the input space R n to a Voronoi Diagram with centers { ˜ c 1 , ..., ˜ c K } given by ˜ c
Revised: The linear classifier parameterized by W , b partitions the input space R n to a Voronoi Diagram with centers { ˜ c 1 , ..., ˜ c K } given by ˜ c

Original: k = 12 W k if b k = − 

Revised: k = 12 W k if b k = −

Original: || W k
Revised: || W k

Original: || 22 , k = 1 , ..., K .
Revised: || 22 , k = 1 , ..., K .

Original: Proof.
Revised: Proof.

Original: We ﬁrst articulate Lemma E.1 and ﬁnd the exact relationship between the hyperplane Π k ( z ) and the center of its associated cell in R n .
Revised: We first articulate Lemma E.1 and find the exact relationship between the hyperplane Π k ( z ) and the center of its associated cell in R n .

Original: By Deﬁnition 3.1, the cell for a point z ∈ R n is found by comparing d ( z , c k ) 2 − ν k for different k , so we deﬁne the power function p ( z , S ) expressing this value
Revised: By Definition 2.1, the cell for a point z ∈ R n is found by comparing d ( z , c k ) 2 − ν k for different k , so we define the power function p ( z , S ) expressing this value

Original: [Equation]
Revised: [Equation]

Original: in which S ⊆ R n is a sphere with center u and radius r .
Revised: in which S ⊆ R n is a sphere with center u and radius r .

Original: In fact, the weight ν associated with a center in Deﬁnition 3.1 can be interpreted as the square of the radius r 2 .
Revised: In fact, the weight ν associated with a center in Definition 2.1 can be interpreted as the square of the radius r 2 .

Original: Next, let U denote a paraboloid y =
Revised: Next, let U denote a paraboloid y =

Original: z 2 , let Π( S ) be the transform that maps sphere S with center u and radius r into hyperplane
Revised: z 2 , let Π( S ) be the transform that maps sphere S with center u and radius r into hyperplane

Original: [Equation]
Revised: [Equation]

Original: It can be proved that Π is a bijective mapping between arbitrary spheres in R n and nonvertical hyperplanes in R n +1 that intersect U (Aurenhammer, 1987).
Revised: It can be proved that Π is a bijective mapping between arbitrary spheres in R n and nonvertical hyperplanes in R n +1 that intersect U (Aurenhammer, 1987).

Original: Further, let z (cid:48) denote the vertical projection of z onto U and z (cid:48)(cid:48) denote its vertical projection onto Π( S ) , then the power function can be written as
Revised: Further, let z ′ denote the vertical projection of z onto U and z ′′ denote its vertical projection onto Π( S ) , then the power function can be written as

Original: [Equation]
Revised: [Equation]

Original: which implies the following relationships between a sphere in R n and an associated hyperplane in R n +1 (Lemma 4 in (Aurenhammer, 1987)): let S 1 and S 2 be nonco-centeric spheres in R n , then the bisector of their Power cells is the vertical projection of Π( S 1 )
Revised: which implies the following relationships between a sphere in R n and an associated hyperplane in R n +1 (Lemma 4 in (Aurenhammer, 1987)): let S 1 and S 2 be nonco-centeric spheres in R n , then the bisector of their Power cells is the vertical projection of Π( S 1 )

Original: ∩ Π( S 2 ) onto R n .
Revised: ∩ Π( S 2 ) onto R n .

Original: Now, we have a direct relationship between sphere S , and hyperplane Π( S ) , and comparing equation (3) with the hyperplanes used in logistic regression { Π k ( z ) :
Revised: Now, we have a direct relationship between sphere S , and hyperplane Π( S ) , and comparing equation (3) with the hyperplanes used in logistic regression { Π k ( z ) :

Original: W Tk z + b k } Kk =1 gives us
Revised: W Tk z + b k } Kk =1 gives us

Original: [Equation]
Revised: [Equation]

Original: Although there is no guarantee that b k + 14 || W k
Revised: Although there is no guarantee that b k + 14 || W k

Original: || 22 is always positive for an arbitrary logistic regression model, we can impose a constraint on r 2 to keep it be zero during the optimization, which implies
Revised: || 22 is always positive for an arbitrary logistic regression model, we can impose a constraint on r 2 to keep it be zero during the optimization, which implies

Original: [Equation]
Revised: [Equation]

Original: By this way, the radii for all K spheres become identical (all zero).
Revised: By this way, the radii for all K spheres become identical (all zero).

Original: After the optimization of logistic regression model, the centers { 12 W k } Kk =1 will be used as probing-induced Voronoi centers.
Revised: After the optimization of logistic regression model, the centers { 12 W k } Kk =1 will be used as probing-induced Voronoi centers.

Original: F I MPLEMENTATION D ETAILS AND R ESULT A NALYSIS OF C OMPREHENSIVE A BLATION S TUDIES
Revised: F I MPLEMENTATION D ETAILS AND R ESULT A NALYSIS OF C OMPREHENSIVE A BLATION S TUDIES

Original: iVoro.
Revised: iVoro.

Original: We generally follow the protocol of PASS (Zhu et al., 2021) to train the feature extractor φ but only on the data from the ﬁrst phase, i.e. 50 (for 5/10 phases) or 40 (for 20 phases) classes of
Revised: We generally follow the protocol of PASS (Zhu et al., 2021) to train the feature extractor ϕ but only on the data from the first phase, i.e. 50 (for 5/10 phases) or 40 (for 20 phases) classes of

Original: CIFAR-100, 100 classes of TinyImageNet, and 50 classes of ImageNet-Subset.
Revised: CIFAR-100, 100 classes of TinyImageNet, and 50 classes of ImageNet-Subset.

Original: We also reproduce the results of PASS, using the same hyper-parameters, e.g. the weight for the knowledge distillation loss set at 10, and the weight for the prototype augmentation loss set at 10.
Revised: We also reproduce the results of PASS, using the same hyper-parameters, e.g. the weight for the knowledge distillation loss set at 10, and the weight for the prototype augmentation loss set at 10.

Original: For iVoro (and all its subsequent variants), the trained model is frozen after the ﬁrst phase and throughout all the remaining phases.
Revised: For iVoro (and all its subsequent variants), the trained model is frozen after the first phase and throughout all the remaining phases.

Original: At phase t , the prototypical centers { c } are computed for both the current and the historical phases τ ∈ { 1 , ..., t } , and are used to construct the Voronoi Diagram.
Revised: At phase t , the prototypical centers { c } are computed for both the current and the historical phases τ ∈ { 1 , ..., t } , and are used to construct the Voronoi Diagram.

Original: The simplest iVoro method (i.e. the vanilla Voronoi Diagram, or 1-nearest-neighbor) can already achieve comparable or even better results than the state-of-the-art non-exemplar CIL methods.
Revised: The simplest iVoro method (i.e. the vanilla Voronoi Diagram, or 1-nearest-neighbor) can already achieve comparable or even better results than the state-of-the-art non-exemplar CIL methods.

Original: For example, the difference in accuracy in comparison to PASS is 0.29%/6.91%/3.63% for 5/10/20phase CIFAR-100, -3.58%/-1.09%/5.43% for 5/10/20-phase TinyImageNet, and 4.76% for 10-phase
Revised: For example, the difference in accuracy in comparison to PASS is 0.29%/6.91%/3.63% for 5/10/20phase CIFAR-100, -3.58%/-1.09%/5.43% for 5/10/20-phase TinyImageNet, and 4.76% for 10-phase

Original: ImageNet-Subset.
Revised: ImageNet-Subset.

Original: Notably, there is always a signiﬁcant elevation of accuracy on long-phase data, suggesting the continuous ﬁne-tuning of model, even with improved loss functions, tends to forget seriously on earlier data.
Revised: Notably, there is always a significant elevation of accuracy on long-phase data, suggesting the continuous fine-tuning of model, even with improved loss functions, tends to forget seriously on earlier data.

Original: With a ﬁxed feature extractor, iVoro has shown an improved ability to overcome catastrophic forgetting.
Revised: With a fixed feature extractor, iVoro has shown an improved ability to overcome catastrophic forgetting.

Original: On the other hand, for short-phase data, iVoro is similar or worse than the state-of-the-art method, probably because of the prototypical centers are computed without considering data distribution (i.e. simply the mean of features).
Revised: On the other hand, for short-phase data, iVoro is similar or worse than the state-of-the-art method, probably because of the prototypical centers are computed without considering data distribution (i.e. simply the mean of features).

Original: iVoro-N. To inspect the effectiveness of the parameterized feature transformation, we apply Lnormalization with/without Tukey’s ladder of powers transformation ( λ varying from 0.3 to 0.9), and compare with iVoro.
Revised: iVoro-N. To inspect the effectiveness of the parameterized feature transformation, we apply Lnormalization with/without Tukey’s ladder of powers transformation ( λ varying from 0.3 to 0.9), and compare with iVoro.

Original: Generally, the improvement acquired from the feature normalization is more prominent on more complex datasets (e.g. TinyImageNet and ImageNet-Subset), or simpler datasets with longer phases (e.g. CIFAR-100 with 20 phases), with improvements ranging from 1.65% to 2.40% higher than iVoro.
Revised: Generally, the improvement acquired from the feature normalization is more prominent on more complex datasets (e.g. TinyImageNet and ImageNet-Subset), or simpler datasets with longer phases (e.g. CIFAR-100 with 20 phases), with improvements ranging from 1.65% to 2.40% higher than iVoro.

Original: The detailed analysis is presented in Sec.
Revised: The detailed analysis is presented in Sec.

Original: H. iVoro-D/iVoro-ND.
Revised: H. iVoro-D/iVoro-ND.

Original: The detailed algorithm of iVoro-D is presented in Alg. 3.
Revised: The detailed algorithm of iVoro-D is presented in Alg. 3.

Original: Speciﬁcally, for each phase τ ∈ { 1 , ..., t } , the local dataset D τ is used to train a logistic regression model (restricted by Thm. 3.1) with weight decay β at 0.0001 and initial learning rate at 0.001.
Revised: Specifically, for each phase τ ∈ { 1 , ..., t } , the local dataset D τ is used to train a logistic regression model (restricted by Thm. 2.1) with weight decay β at 0.0001 and initial learning rate at 0.001.

Original: The result is also shown in Tab. 2.
Revised: The result is also shown in Tab. 2.

Original: Aided by the D&C algorithm and local logistic regression, iVoro-D is consistently better than iVoro, e.g. 0.48% ∼ 0.96% higher on the CIFAR-100 dataset, 1.75% ∼ 3.44% higher on the TinyImageNet dataset, and 0.72% higher on the ImageNet-Subset dataset.
Revised: Aided by the D&C algorithm and local logistic regression, iVoro-D is consistently better than iVoro, e.g. 0.48% ∼ 0.96% higher on the CIFAR-100 dataset, 1.75% ∼ 3.44% higher on the TinyImageNet dataset, and 0.72% higher on the ImageNet-Subset dataset.

Original: When further combined with feature normalization, iVoro-ND achieves even higher accuracy, 54.72% on 20-phase
Revised: When further combined with feature normalization, iVoro-ND achieves even higher accuracy, 54.72% on 20-phase

Original: CIFAR-100, 42.10% on 20-phase TinyImageNet, and 58.52% on ImageNet-Subset.
Revised: CIFAR-100, 42.10% on 20-phase TinyImageNet, and 58.52% on ImageNet-Subset.

Original: For comparison,
Revised: For comparison,

Original: PASS reaches 48.75% on 20-phase CIFAR-100, 32.86% on 20-phase TinyImageNet, and 50.63% on
Revised: PASS reaches 48.75% on 20-phase CIFAR-100, 32.86% on 20-phase TinyImageNet, and 50.63% on

Original: ImageNet-Subset.
Revised: ImageNet-Subset.

Original: Therefore without incorporating more sophisticate techniques like iVoro-R/iVoroAC/iVoro-L, the iVoro-ND method can already surpass previous state-of-the-art method by a large margin e.g. 5.97%/9.24%/7.89%.
Revised: Therefore without incorporating more sophisticate techniques like iVoro-R/iVoroAC/iVoro-L, the iVoro-ND method can already surpass previous state-of-the-art method by a large margin e.g. 5.97%/9.24%/7.89%.

Original: iVoro-AC/iVoro-AI/iVoro-NAC/iVoro-NAI.
Revised: iVoro-AC/iVoro-AI/iVoro-NAC/iVoro-NAI.

Original: While the previous variants of iVoro only consider the prediction on the original image/class, here we show that the prediction can be substantially improved by augmentation consensus (iVoro-AC) and augmentation integration (iVoro-AI) proposed in this paper.
Revised: While the previous variants of iVoro only consider the prediction on the original image/class, here we show that the prediction can be substantially improved by augmentation consensus (iVoro-AC) and augmentation integration (iVoro-AI) proposed in this paper.

Original: Speciﬁcally, iVoro-AC improves upon iVoro by 13.76% ∼ 14.58% on CIFAR-100, by 16.99% ∼ 17.00% on TinyImageNet, and by 16.50% on ImageNet-Subset.
Revised: Specifically, iVoro-AC improves upon iVoro by 13.76% ∼ 14.58% on CIFAR-100, by 16.99% ∼ 17.00% on TinyImageNet, and by 16.50% on ImageNet-Subset.

Original: iVoro-AI itself generally works worse than iVoro-AC, e.g. improves up to 6.73% on CIFAR-100, up to 9.82% on TinyImageNet, and 5.26% on ImageNet-Subset, but if combined with feature normalization, iVoro-NAI performs much better than iVoro-NAC on TinyImageNet (up to 65.83%) and ImageNet-Subset (77.12%).
Revised: iVoro-AI itself generally works worse than iVoro-AC, e.g. improves up to 6.73% on CIFAR-100, up to 9.82% on TinyImageNet, and 5.26% on ImageNet-Subset, but if combined with feature normalization, iVoro-NAI performs much better than iVoro-NAC on TinyImageNet (up to 65.83%) and ImageNet-Subset (77.12%).

Original: When augmentation consensus/integration is used, the previous variants iVoro-N and iVoro-D can all be promoted.
Revised: When augmentation consensus/integration is used, the previous variants iVoro-N and iVoro-D can all be promoted.

Original: Generally, adding an additional component will bring in more performance gain, as shown in Tab. 2 in detail.
Revised: Generally, adding an additional component will bring in more performance gain, as shown in Tab. 2 in detail.

Original: iVoro-NACL/iVoro-NAIL/iVoro-NDAIL.
Revised: iVoro-NACL/iVoro-NAIL/iVoro-NDAIL.

Original: We further validate the multilayer VD for multiple feature spaces.
Revised: We further validate the multilayer VD for multiple feature spaces.

Original: As a proof of concept, we only extract the feature from the third block to build an additional
Revised: As a proof of concept, we only extract the feature from the third block to build an additional

Original: VD and conduct the integration using CCVD (Def. 3.3) with γ set at 1.
Revised: VD and conduct the integration using CCVD (Def. 2.3) with γ set at 1.

Original: Compared with iVoroNAC, multilayer VD (iVoro-NACL) further improves both average accuracy and last accuracy on
Revised: Compared with iVoroNAC, multilayer VD (iVoro-NACL) further improves both average accuracy and last accuracy on

Original: CIFAR-100 under 5/10 phases settings (2.36%/2.20% better on last phase, respectively), which also realizes the highest performance on the given settings among all ablation settings.
Revised: CIFAR-100 under 5/10 phases settings (2.36%/2.20% better on last phase, respectively), which also realizes the highest performance on the given settings among all ablation settings.

Original: The ﬁnal performance on ImageNet-Subset is also improved by 2.52%.
Revised: The final performance on ImageNet-Subset is also improved by 2.52%.

Original: In the meanwhile, multilayer VD does not make obvious difference on TinyImageNet.
Revised: In the meanwhile, multilayer VD does not make obvious difference on TinyImageNet.

Original: When adding multilayer VD to iVoro-NAI, labelled as iVoro-NAIL, signiﬁcant performance gain is observed on all experiments of CIFAR- (9.1%/9.1%/15.76% average accuracy increments on 5/10/20 phases) and large improvement is also achieved on ImageNet-Subset (6.06% ﬁnal accuracy growth).
Revised: When adding multilayer VD to iVoro-NAI, labelled as iVoro-NAIL, significant performance gain is observed on all experiments of CIFAR- (9.1%/9.1%/15.76% average accuracy increments on 5/10/20 phases) and large improvement is also achieved on ImageNet-Subset (6.06% final accuracy growth).

Original: On the contrary, only limited gain is presented on TinyImageNet.
Revised: On the contrary, only limited gain is presented on TinyImageNet.

Original: The above ablation results may demonstrate that multilayer VD with augmentation integration works extraordinarily well for small class set (100 classes), but not very effective when class number increases.
Revised: The above ablation results may demonstrate that multilayer VD with augmentation integration works extraordinarily well for small class set (100 classes), but not very effective when class number increases.

Original: Robustness Analysis.
Revised: Robustness Analysis.

Original: We run PASS 5 times on CIFAR-100 with the 10-phase setting, and the last accuracies (%) are 48.25%, 49.03%, 53.03%, 53.95%, and 54.75%, respectively, with mean and standard deviation (std) being 51.80% ± 2.65%; meanwhile, we also test PASS for 5 runs on
Revised: We run PASS 5 times on CIFAR-100 with the 10-phase setting, and the last accuracies (%) are 48.25%, 49.03%, 53.03%, 53.95%, and 54.75%, respectively, with mean and standard deviation (std) being 51.80% ± 2.65%; meanwhile, we also test PASS for 5 runs on

Original: ImageNet-Subset with 10 phases, and the last accuracies (%) are 49.85%, 50.63%, 51.03%, 51.88%, and 52.52%, respectively, with mean ± std being 51.18% ± 0.94%.
Revised: ImageNet-Subset with 10 phases, and the last accuracies (%) are 49.85%, 50.63%, 51.03%, 51.88%, and 52.52%, respectively, with mean ± std being 51.18% ± 0.94%.

Original: As shown above, even though
Revised: As shown above, even though

Original: PASS achieves relatively good accuracy on average, the training is not very stable on CIFAR-100, where the difference between the highest and lowest accuracy on CIFAR-100 is as large as 6.5%; while the performance on ImageNet-Subset is slightly better regarding the robustness, PASS still ranges from 49.85% to 52.52%.
Revised: PASS achieves relatively good accuracy on average, the training is not very stable on CIFAR-100, where the difference between the highest and lowest accuracy on CIFAR-100 is as large as 6.5%; while the performance on ImageNet-Subset is slightly better regarding the robustness, PASS still ranges from 49.85% to 52.52%.

Original: On the contrary, compared to PASS, our iVoro method is naturally robust to various datasets with no ﬂuctuation in performance, due to the frozen feature extractor and unbiased classiﬁer based on VD.
Revised: On the contrary, compared to PASS, our iVoro method is naturally robust to various datasets with no fluctuation in performance, due to the frozen feature extractor and unbiased classifier based on VD.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: G A DDITIONAL F IGURES AND T ABLES
Revised: G A DDITIONAL F IGURES AND T ABLES

Original: [Table]
Revised: [Table]

Original: 83.18 (cid:70)

Original: [-♠♣ (cid:7)(cid:72) +9.
-]
Revised: [-♠♣ (cid:7)(cid:72) +9.
-]

Original: [-+10.
-]
Revised: [-+10.
-]

Original: [-+9.10 +9.
-]
Revised: [-+9.10 +9.
-]

Original: [-+15.76 +13.
-]
Revised: [-+15.76 +13.
-]

Original: [-+1.
-]
Revised: [-+1.
-]

Original: [-+1.
-]
Revised: [-+1.
-]

Original: [-+1.
-]
Revised: [-+1.
-]

Original: +1.47 +1.26 +1.54 +3.

Original: [Figure]

Original: iVoro

Original: [Figure]

Original: NDAIL 77.57 66.54 72.50 62.28 81.24 71.45 81.74 72.34 80.22 71.13 79.08 69.95 90.

Original: [Figure]
Revised: [Figure]

Original: 83.84 (cid:70)

Original: ♠♣ (cid:7)(cid:72) +5.95 +5.

Original: [Figure]

Original: 2.63 +1.

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: 0.2 Accuracy 0.
Revised: 0.2 Accuracy 0.

Original: E n t r o p y b a s e d
Revised: E n t r o p y b a s e d

Original: G e o m e t r i c V a r i a n c e 1e 4 Pearson's R: -0.378 0.701 (A) CIFAR-100 (5 phases) 0.
Revised: G e o m e t r i c V a r i a n c e 1e 4 Pearson's R: -0.378 0.701 (A) CIFAR-100 (5 phases) 0.

Original: 0.6 Accuracy 

Revised: 0.6 Accuracy

Original: E n t r o p y b a s e d
Revised: E n t r o p y b a s e d

Original: G e o m e t r i c V a r i a n c e 1e 5 0.042 0.617 (B) TinyImageNet (5 phases) 0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 Accuracy 0.
Revised: G e o m e t r i c V a r i a n c e 1e 5 0.042 0.617 (B) TinyImageNet (5 phases) 0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 Accuracy 0.

Original: E n t r o p y b a s e d
Revised: E n t r o p y b a s e d

Original: G e o m e t r i c V a r i a n c e 1e 4 Pearson's R: -0.927 0.906 (C) ImageNet-Subset (10 phases)
Revised: G e o m e t r i c V a r i a n c e 1e 4 Pearson's R: -0.927 0.906 (C) ImageNet-Subset (10 phases)

Original: Each of the rotated images can possibly be assigned to each of the expanded labels (i.e. teddy (1) , teddy (2) , teddy (3) , and teddy (4) ).

Original: The ﬁnal score of "teddy" will be the aggregation of all thepredictions.

Original: The variance of the 16 predictions reﬂects the conﬁdence of this prediction.

Original: H D ETAILED A NALYSIS ON P ARAMETERIZED F EATURE N ORMALIZATION
Revised: H D ETAILED A NALYSIS ON P ARAMETERIZED F EATURE N ORMALIZATION

Original: In this section we give a detailed comparison between iVoro (no normalization) and iVoro-N (parameterized feature normalization).
Revised: In this section we give a detailed comparison between iVoro (no normalization) and iVoro-N (parameterized feature normalization).

Original: On both CIFAR-100 and TinyImageNet with different phases, we show the distribution of accuracy (across all phases) for iVoro, iVoro-N with only L 2 normalization, and iVoro-N with both L 2 normalization and Tukey’s ladder of powers transformation ( λ varying from 0.3 to 0.9).
Revised: On both CIFAR-100 and TinyImageNet with different phases, we show the distribution of accuracy (across all phases) for iVoro, iVoro-N with only L 2 normalization, and iVoro-N with both L 2 normalization and Tukey’s ladder of powers transformation ( λ varying from 0.3 to 0.9).

Original: As shown in Fig.
Revised: As shown in Fig.

Original: H.1, feature normalization beneﬁts both datasets, but the efﬁcacy is more prominent in more complex dataset e.g. TinyImageNet.
Revised: H.1, feature normalization benefits both datasets, but the efficacy is more prominent in more complex dataset e.g. TinyImageNet.

Original: Overall, iVoro-N improves the accuracy in the last phase by up to 1.65% on CIFAR-100, 2.21% on TinyImageNet, and 2.40 on
Revised: Overall, iVoro-N improves the accuracy in the last phase by up to 1.65% on CIFAR-100, 2.21% on TinyImageNet, and 2.40 on

Original: ImageNet-Subset, respectively, compared to iVoro.
Revised: ImageNet-Subset, respectively, compared to iVoro.

Original: [Figure]
Revised: [Figure]

Original:  [Figure]
Revised: TinyImageNet [Figure] [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: TinyImageNet 
Revised: TinyImageNet (10 phases)

Original: [Figure]
Revised: [Figure]

Original: [Figure] 
Revised: [Figure] [Figure] [Figure] N(

Original: [Figure] 
Revised: [Figure] [Figure] =0.4) iVoro-N(L2, =0.5)

Original: [Figure]
Revised: iVoro-N(L2, =0.6) iVoro-N(L2, =0.7) iVoro-N(L2, =0.8) iVoro-N(L2, =0.9) iVoro-N(L2)

Original: [Figure]
Revised: [Figure]

Original: I U NCERTAINTY A NALYSIS
Revised: I U NCERTAINTY A NALYSIS

Original: In this section
Revised: In this section

Original: [Figure]
Revised: [Figure]

Original: we demonstrate that the Entropy-based geometric variance (HV) is a good indicator of predictive uncertainty induced by SSL-based label augmentation
Revised: we demonstrate that the Entropy-based geometric variance (HV) is a good indicator of predictive uncertainty induced by SSL-based label augmentation

Original: [Figure]
Revised: [Figure]

Original: and also exhibits high correlation with predictive accuracy.
Revised: and also exhibits high correlation with predictive accuracy.

Original: Fig.
Revised: Fig.

Original: I.1, Fig.
Revised: I.1, Fig.

Original: I.2 and Fig.
Revised: I.2 and Fig.

Original: I.3 present the distribution of HV of each class during each phase (0 to 5) in 5-phase CIFAR-100 data.
Revised: I.3 present the distribution of HV of each class during each phase (0 to 5) in 5-phase CIFAR-100 data.

Original: The color of the box reﬂects the accuracy of iVoro for the speciﬁc class.
Revised: The color of the box reflects the accuracy of iVoro for the specific class.

Original: It can be observed in these ﬁgures that, the class with higher predictive accuracy will typically exhibit lower geometric uncertainty
Revised: It can be observed in these figures that, the class with higher predictive accuracy will typically exhibit lower geometric uncertainty.

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure] 
Revised: [Figure] [Table] [Figure] [Table]

Original:  [Table]
Revised: [Figure] [Table] [Figure] [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table] [Figure]
Revised: [Table] [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: E n t r o p y b a s e d
Revised: E n t r o p y b a s e d

Original: G e o m e t r i c
Revised: G e o m e t r i c

Original: V a r i a n c e
Revised: V a r i a n c e

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: E n t r o p y b a s e d
Revised: E n t r o p y b a s e d

Original: G e o m e t r i c
Revised: G e o m e t r i c

Original: V a r i a n c e
Revised: V a r i a n c e

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: J A NALYSIS ON THE F EATURE E XTRACTOR
Revised: J A NALYSIS ON THE F EATURE E XTRACTOR

Original: In order to examine the effect of the feature extractor on the ﬁnal result
Revised: In order to examine the effect of the feature extractor on the final result

Original: [Figure]
Revised: [Figure]

Original: we gradually decrease the number of classes in the ﬁrst phase from 50 to 40, 30, 20, and 10 and still include 5 classes in each subsequent phase.
Revised: we gradually decrease the number of classes in the first phase from 50 to 40, 30, 20, and 10 and still include 5 classes in each subsequent phase.

Original: When compared with PASS, the best version of iVoro still has 17.75%, 13.59%, 10.89%
Revised: When compared with PASS, the best version of iVoro still has 17.75%, 13.59%, 10.89%

Original: [Figure]
Revised: [Figure]

Original: and 1.60% improvements with 40, 30,
Revised: and 1.60% improvements with 40, 30,

Original: [Figure]
Revised: [Figure]

Original: and 10 initial classes, respectively
Revised: and 10 initial classes, respectively

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: K A NALYSIS ON THE M ULTILAYER V ORONOI D IAGRAMS
Revised: K A NALYSIS ON THE M ULTILAYER V ORONOI D IAGRAMS

Original: In this section, we extract the feature from the 3 rd block and rebuild iVoro, to be iVoro3, etc.
Revised: In this section, we extract the feature from the 3 rd block and rebuild iVoro, to be iVoro3, etc.

Original: The results for iVoro3, iVoro3-N, iVoro3-NAC, and iVoro3-NAI are shown in Fig.
Revised: The results for iVoro3, iVoro3-N, iVoro3-NAC, and iVoro3-NAI are shown in Fig.

Original: F.2, Fig.
Revised: F.2, Fig.

Original: F.3, and
Revised: F.3, and

Original: Fig.
Revised: Fig.

Original: F.1.
Revised: F.1.

Original: As expected, there is always a substantial decrease in accuracy if only the features from the 3 rd block are used.
Revised: As expected, there is always a substantial decrease in accuracy if only the features from the 3 rd block are used.

Original: For example, iVoro drops from 38.27% to 18.71%, from 56.05% to 43.22%, and from 55.40% to 35.10% on TinyImageNet, CIFAR-100, and ImageNet-Subset, respectively.
Revised: For example, iVoro drops from 38.27% to 18.71%, from 56.05% to 43.22%, and from 55.40% to 35.10% on TinyImageNet, CIFAR-100, and ImageNet-Subset, respectively.

Original: However, when augmentation integration is used, the accuracy of iVoro3-NAI becomes 42.03% (TinyImageNet), 66.13% (CIFAR-100), and 64.32% (ImageNet-Subset), even much higher than iVoro and PASS.
Revised: However, when augmentation integration is used, the accuracy of iVoro3-NAI becomes 42.03% (TinyImageNet), 66.13% (CIFAR-100), and 64.32% (ImageNet-Subset), even much higher than iVoro and PASS.

Original: Moreover, when integrating the features from φ and φ (3) using CCVD, iVoro-NDAIL achieves the best performance across all variants of iVoro, 72.34% on TinyImageNet and 83.84% on
Revised: Moreover, when integrating the features from ϕ and ϕ (3) using CCVD, iVoro-NDAIL achieves the best performance across all variants of iVoro, 72.34% on TinyImageNet and 83.84% on

Original: ImageNet-Subset.
Revised: ImageNet-Subset.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: CIFAR-100 (5 phases)
Revised: CIFAR-100 (5 phases)

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: L C OMPLETE L IST OF A LGORITHMS
Revised: L C OMPLETE L IST OF A LGORITHMS

Original: Here we provide the algorithm for Voronoi Diagram reduction (Ma et al., 2022a) in Alg. 1, the iVoro algorithm in Alg. 2, the iVoro-D algorithm in Alg. 3.
Revised: Here we provide the algorithm for Voronoi Diagram reduction (Ma et al., 2022a) in Alg. 1, the iVoro algorithm in Alg. 2, the iVoro-D algorithm in Alg. 3.

Original: Algorithm 1: Voronoi Diagram-based Logistic Regression.
Revised: Algorithm 1: Voronoi Diagram-based Logistic Regression.

Original: Data: Local data D t in phase τ
Revised: Data: Local data D t in phase τ

Original: Result: W τ 1 Initialize W τ ← W (0) τ ; 2 for epoch ← 1 , ..., #epoch do b τ,k ← − 14 || W τ,k || 22 , ∀ k
Revised: Result: W τ 1 Initialize W τ ← W (0) τ ; 2 for epoch ← 1 , ..., #epoch do b τ,k ← − 14 || W τ,k || 22 , ∀ k

Original: = 1 , ..., K τ ; (cid:47) Apply Theorem 3.
Revised: = 1 , ..., K τ ; ◁ Apply Theorem 2.

Original: Compute loss L ( W τ , b τ ) ; (cid:47) forward propagation
Revised: Compute loss L ( W τ , b τ ) ; ◁ forward propagation

Original: Update W τ ; (cid:47) backward propagation 6 end 7 return W τ
Revised: Update W τ ; ◁ backward propagation 6 end 7 return W τ

Original: Algorithm 2: iVoro Algorithm.
Revised: Algorithm 2: iVoro Algorithm.

Original: Data: Training datasets until phase t :
Revised: Data: Training datasets until phase t :

Original: D τ = { ( x τ,i , y τ,i ) }
Revised: D τ = { ( x τ,i , y τ,i ) }

Original: N τ i =1 , x τ,i ∈ D , y τ,i ∈
Revised: N τ i =1 , x τ,i ∈ D , y τ,i ∈

Original: C τ , τ ∈ { 1 , ..., t } , query example x
Revised: C τ , τ ∈ { 1 , ..., t } , query example x

Original: Result: prediction ˆ y 1 for τ ∈ { 1 , ..., t } do for k ∈ { 1 , ..., K τ } do c τ,k ← 1 N τ,k (cid:80) i ∈{ 1 ,...,N τ,k } ,y = 
Revised: Result: prediction ˆ y 1 for τ ∈ { 1 , ..., t } do for k ∈ { 1 , ..., K τ } do c τ,k ← 1 N τ,k (cid:80) i ∈{ 1 ,...,N τ,k } ,y = k ϕ

Original: k φ ( x τ,i ) ; (cid:47) prototypical centers ν τ,k ← end 6 end 7 z ← φ ( x ) 8 z ← ( h λ ◦ g w,η ◦ f )( z ) ; (cid:47) iVoro-N, optional  ˆ y ← C τ (cid:48) ,k (cid:48) s.t.
Revised: ( x τ,i ) ; ◁ prototypical centers ν τ,k ← end 6 end 7 z ← ϕ ( x ) 8 z ← ( h λ ◦ g w,η ◦ f )( z ) ; ◁ iVoro-N, optional 9 ˆ y ← C τ ′ ,k ′ s.t.

Original: d ( z , c τ (cid:48) ,k (cid:48) )
Revised: d ( z , c τ ′ ,k ′ )

Original: =
Revised: =

Original: min τ,k d ( z , c τ,k ) 10 return { c τ,k } , ˆ y
Revised: min τ,k d ( z , c τ,k ) 10 return { c τ,k } , ˆ y

Original: D τ = { ( x τ,i , y τ,i ) }
Revised: D τ = { ( x τ,i , y τ,i ) }

Original: N τ i =1 , x τ,i ∈ D , y τ,i ∈
Revised: N τ i =1 , x τ,i ∈ D , y τ,i ∈

Original: C τ , τ ∈ { 1 , ..., t } , query example x
Revised: C τ , τ ∈ { 1 , ..., t } , query example x

Original: Result:
Revised: Result:

Original: Prediction ˆ y 1 for τ ∈ { 1 , ..., t } do
Revised: Prediction ˆ y 1 for τ ∈ { 1 , ..., t } do

Original: W τ , b τ ← Algorithm 1 ( D τ ) ˜ c τ,k ← 12 W τ,k ;
Revised: W τ , b τ ← Algorithm 1 ( D τ ) ˜ c τ,k ← 12 W τ,k ;

Original: c τ,k ← do v← ˜
Revised: c τ,k ← do v ˜

Revised: [Equation]

Revised: [Equation]

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original:  q ← || ˜ c
Revised: | q ← || ˜ c

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original:  22 −
Revised: | 22 −|

Original: [Equation]
Revised: [Equation]

Original: ˜
Revised: ˜

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [-| 
-]
Revised: [-| 
-]

Original: [Equation]
Revised: [Equation]

Original: ˜ 
Revised: ˜ c

Original: [Equation]
Revised: [Equation]

Original: ,k 
Revised: ,k 1 − ˜ c

Original: [Equation]
Revised: [Equation]

Original: ˜ c

Original: [Equation]
Revised: [Equation]

Original: ,k 2 || 2 ;
Revised: 2 || 2 ;

Original: end 9 end 10 for  τ 1 ,
Revised: end 9 end 10 for C τ 1 ,

Original: C τ 1 , C τ 2 ,
Revised: C τ 1 , C τ 2 ,k ′

Original: (cid:48) C τ 2 do v← c τ 1 ,k − c τ 2 ,k (cid:48)
Revised: C τ 2 do v

Original: || c τ 1 ,k − c τ 2 ,k (cid:48) ||
Revised: [Equation]

Original: q ← || [-c τ 1 ,k || 
-]
Revised: q ← || [-c τ 1 ,k || 
-][Equation]

Original: −|| c τ 2 ,k (cid:48)
Revised: [Equation]

Original: ||
Revised: [Equation] [Equation] [Equation]

Original: 22 2 || c τ 1 ,k − c τ 2 ,k (cid:48)
Revised: | c τ 1 , [Equation] − c [Equation]

Revised: [Equation]

Original: || 2 ;
Revised: || 2 ;

Original: 13 end  for do 
Revised: 13 end 14 for do end 19 end 20 return ˆ y

Original: M F ORGETTING A NALYSIS
Revised: M F ORGETTING A NALYSIS

Original: Following PASS (Zhu et al., 2021), we also quantitatively measure the degree of the catastrophic forgetting.
Revised: Following PASS (Zhu et al., 2021), we also quantitatively measure the degree of the catastrophic forgetting.

Original: Speciﬁcally, at phase t , the accuracy drop from the maximum accuracy to current accuracy for datasets from each phase τ ∈ { 1 , ..., t } is denoted as the average forgetting, as shown in Fig.
Revised: Specifically, at phase t , the accuracy drop from the maximum accuracy to current accuracy for datasets from each phase τ ∈ { 1 , ..., t } is denoted as the average forgetting, as shown in Fig.

Original: M. and Fig.
Revised: M. and Fig.

Original: M.2 for CIFAR-100 and TinyImageNet, respectively.
Revised: M.2 for CIFAR-100 and TinyImageNet, respectively.

Original: For PASS, the average forgetting keeps growing during phases.
Revised: For PASS, the average forgetting keeps growing during phases.

Original: However, for iVoro/iVoro-N/iVoro-NAC/iVoro-NAI, the catastrophic forgetting is signiﬁcantly overcome.
Revised: However, for iVoro/iVoro-N/iVoro-NAC/iVoro-NAI, the catastrophic forgetting is significantly overcome.

Original: For example, on CIFAR-100 (5 phase), forgetting in the last phase is decreased from 20.28 to 8.17/8.95/5.92 for iVoro/iVoro-N/iVoro-NAC.
Revised: For example, on CIFAR-100 (5 phase), forgetting in the last phase is decreased from 20.28 to 8.17/8.95/5.92 for iVoro/iVoro-N/iVoro-NAC.

Original: For a local dataset D τ = { ( x τ,i , y τ,i ) } N τ i =1 , x τ,i ∈ D , y τ,i ∈ C τ and a set of ﬁxed prototypes { c τ,k } τ ∈{ 1 ,...,t } ,k ∈{ 1 ,...,K τ } , the prediction ˆ
Revised: For a local dataset D τ = { ( x τ,i , y τ,i ) } N τ i =1 , x τ,i ∈ D , y τ,i ∈ C τ and a set of fixed prototypes { c τ,k } τ ∈{ 1 ,...,t } ,k ∈{ 1 ,...,K τ } , the prediction ˆ

Original: y = arg
Revised: y = arg

Original: min k ∈{ 1 ,...,K τ } || φ ( x ) − c τ,k
Revised: min k ∈{ 1 ,...,K τ } || ϕ

Original:  || 22 is less likely to change compared to continuously updated model, as in PASS, implying that, aligning with our intuition (Sec. 1), the VD structure can naturally and successfully combat catastrophic forgetting. 
Revised: ( x ) − c τ,k || 22 is less likely to change compared to continuously updated model, as in PASS, implying that, aligning with our intuition (Sec. 1), the VD structure can naturally and successfully combat catastrophic forgetting. [Figure] [Table] [Figure]

Revised: [Table]

Original: [Figure]
Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Original: [Table] 
Revised: [Table] [Table] [Table] [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Original: [Table] [Table] 
Revised: [Table] [Table] [Table]

Original: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: N C LASS - LEVEL A NALYSIS
Revised: N C LASS - LEVEL A NALYSIS

Original: In order to examine the reason for our strong two-digit improvement ( 25 . 26% , 37 . 09% , and 33 . 21% for CIFAR-100, TinyImageNet
Revised: In order to examine the reason for our strong two-digit improvement ( 25 . 26% , 37 . 09% [Figure] and 33 . 21% for CIFAR-100, TinyImageNet

Original: [Figure]
Revised: [Figure]

Original: and ImageNet-Subset, respectively), in this section (Fig.
Revised: and ImageNet-Subset, respectively), in this section (Fig.

Original: N.1, Fig.
Revised: N.1, Fig.

Original: N.2,
Revised: N.2,

Original: Fig.
Revised: Fig.

Original: N.3, Fig.
Revised: N.3, Fig.

Original: N.4, Fig. N.5, Fig.
Revised: N.4, Fig. N.5, Fig.

Original: N.6, Fig.
Revised: N.6, Fig.

Original: N.
Revised: N.

Original: [Figure]
Revised: [Figure]

Original: and Fig.
Revised: and Fig.

Original: N.8)
Revised: N.8)

Original: [Figure]
Revised: [Figure]

Original: we illustrate the class-level accuracies along the 5 phases for all the 200 classes in TinyImageNet.
Revised: we illustrate the class-level accuracies along the 5 phases for all the 200 classes in TinyImageNet.

Original: Generally, iVoro is competitive to PASS, and iVoro-NAC/NAI performance signiﬁcantly better than the former two.
Revised: Generally, iVoro is competitive to PASS, and iVoro-NAC/NAI performance significantly better than the former two.

Original: This is more evident for novel classes (i
Revised: This is more evident for novel classes (i

Original: [Figure]
Revised: [Figure]

Original: e. class 100-199)
Revised: e. class 100-199)

Original: [Figure]
Revised: [Figure]

Original: showing that augmentation consensus/integration is particularly useful for unseen classes
Revised: showing that augmentation consensus/integration is particularly useful for unseen classes

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original:  [Figure]
Revised: [Table] [Figure] [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Revised: PASSiVoroiVoro

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: O S AMPLE - LEVEL A NALYSIS
Revised: O S AMPLE - LEVEL A NALYSIS

Original: In Sec.
Revised: In Sec.

Original: N, we have analyzed the class-level improvement, and demonstrated that most of the time iVoro-NAI can surpass others by virtue of the postprocessing of the predictions from SSL-based label augmentation.
Revised: N, we have analyzed the class-level improvement, and demonstrated that most of the time iVoro-NAI can surpass others by virtue of the postprocessing of the predictions from SSL-based label augmentation.

Original: In this section, we go into more detail at the sample level aiming at revealing why a misclassiﬁcation can be corrected once our method is applied.
Revised: In this section, we go into more detail at the sample level aiming at revealing why a misclassification can be corrected once our method is applied.

Original: To do so, we arbitrarily inspect two examples from the ﬁrst class (i.e. "umbrella"), and show them in
Revised: To do so, we arbitrarily inspect two examples from the first class (i.e. "umbrella"), and show them in

Original: Fig.
Revised: Fig.

Original: O.1 and Fig.
Revised: O.1 and Fig.

Original: O.2, respectively.
Revised: O.2, respectively.

Original: In Fig.
Revised: In Fig.

Original: O.1, "umbrella" is within the top-3 labels predicted by PASS, which is, however, overwhelmed by "ice_lolly".
Revised: O.1, "umbrella" is within the top-3 labels predicted by PASS, which is, however, overwhelmed by "ice_lolly".

Original: From row 2 to row 9, label set I/II/III/IV represent the expended labels (I for the original, and II/III/IV for the 90/180/270-degree rotated, see Fig.
Revised: From row 2 to row 9, label set I/II/III/IV represent the expended labels (I for the original, and II/III/IV for the 90/180/270-degree rotated, see Fig.

Original: G.4), and the distances from the query image (original or rotated) to the 100 classes (original or expanded) are shown.
Revised: G.4), and the distances from the query image (original or rotated) to the 100 classes (original or expanded) are shown.

Original: Interestingly, the original image is predicted as the same label ("ice_lolly") by iVoro.
Revised: Interestingly, the original image is predicted as the same label ("ice_lolly") by iVoro.

Original: However, for the rotated images, the mis-predicted labels are various, "abacus", "bannister", "limousine", "vestment", "sewing_machine", "plunger", and "dumbbell".
Revised: However, for the rotated images, the mis-predicted labels are various, "abacus", "bannister", "limousine", "vestment", "sewing_machine", "plunger", and "dumbbell".

Original: But importantly, the distance to the ground-truth label "umbrella" is always relatively small.
Revised: But importantly, the distance to the ground-truth label "umbrella" is always relatively small.

Original: Hence, when integrated altogether, the ground-truth label "umbrella" achieves the nearest distance (see upper right ﬁgure).
Revised: Hence, when integrated altogether, the ground-truth label "umbrella" achieves the nearest distance (see upper right figure).

Original: Similarly, in Fig.
Revised: Similarly, in Fig.

Original: O.2, the sample is misclassiﬁed as "beach_wagon" by PASS, and is still misclassiﬁed by iVoro as "volleyball".
Revised: O.2, the sample is misclassified as "beach_wagon" by PASS, and is still misclassified by iVoro as "volleyball".

Original: However, when augmentation integration (iVoro-AI) is applied, the accumulated distances clearly direct to the correct label "umbrella".
Revised: However, when augmentation integration (iVoro-AI) is applied, the accumulated distances clearly direct to the correct label "umbrella".

Original: In conclusion, these sample-level analyses show that iVoro-AC/AI elevates the performance by substantially enhancing the robustness of the (augmented) prediction – the rotated image set and the expanded label set collectively contribute to the strong two-digit improvement.
Revised: In conclusion, these sample-level analyses show that iVoro-AC/AI elevates the performance by substantially enhancing the robustness of the (augmented) prediction – the rotated image set and the expanded label set collectively contribute to the strong two-digit improvement.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]