Original: Uncertainty-Aware Hierarchical Reﬁnement for Incremental Implicitly-Reﬁned Classiﬁcation
Revised: Uncertainty-Aware Hierarchical Reﬁnement for Incremental Implicitly-Reﬁned Classiﬁcation

Original: Abstract
Revised: Abstract

Original: Incremental implicitly-reﬁned classiﬁcation aims at assigning hierarchical labelsto the same instance encountered at different phases.
Revised: Incremental implicitly-reﬁned classiﬁcation aims at assigning hierarchical labelsto the same instance encountered at different phases.

Original: Existing methods tend to failin generating hierarchy-invariant descriptor when novel classes are inherited fromthe old ones.
Revised: Existing methods tend to failin generating hierarchy-invariant descriptor when novel classes are inherited fromthe old ones.

Original: To address the issue, this paper explores the inheritance relations inthe process of multi-level semantic increment, and propose an Uncertainty-Aware Hierarchical Reﬁnement (UAHR) scheme.
Revised: To address the issue, this paper explores the inheritance relationsin the process of multi-level semantic increment, and proposes an UncertaintyAware Hierarchical Reﬁnement (UAHR) scheme.

Original: Our proposed scheme consists of aglobal representation extension strategy that enhances the discrimination of incremental representation by widening the corresponding margin distance, and ahierarchical distribution alignment strategy that reﬁnes the distillation process byexplicitly determining the inheritance relationship of the incremental class.
Revised: Our proposed scheme consistsof a global representation extension strategy that enhances the discrimination ofincremental representation by widening the corresponding margin distance, and ahierarchical distribution alignment strategy that reﬁnes the distillation process byexplicitly determining the inheritance relationship of the incremental class.

Original: Particularly, the shifting subclasses are corrected under the guidance of hierarchicaluncertainty, ensuring the consistency of the homogeneous features.
Revised: Particularly, the shifting subclasses are corrected under the guidance of hierarchicaluncertainty, ensuring the consistency of the homogeneous features.

Original: Extensive experiments on benchmarks IIRC-CIFAR and IIRC-ImageNet-lite demonstrate thesuperiority of our proposed method over state-of-the-art.
Revised: Extensive experiments on benchmarks IIRC-CIFAR and IIRC-ImageNet-lite demonstrate thesuperiority of our proposed method over the state-of-the-art approaches.

Original:  Introduction
Revised:  Introduction

Original: In recent years, deep learning has made huge breakthroughs in the ﬁeld of computer vision, matchingor even surpassing human performance on some  recognition tasks.
Revised: In recent years, deep learning has made huge breakthroughs in the ﬁeld of computer vision, matching or even surpassing human performance on some image recognition tasks.

Original: However, learning multiple tasksin sequence ( i e. , continual learning) remains a major challenge which requires models to have theability to aggregate different learning objectives into a coherent whole over time.
Revised: However, learningmultiple tasks in sequential data ( i e. , continual learning) remains a major challenge, which requiresmodels to have the ability to aggregate different learning objectives into a coherent whole over time.

Original: When a new task comes with the increase of identiﬁed classes ( i e. , class-incremental learning), jointtraining with all old and new data is too time-consuming and labor-intensive.
Revised: When a new task comes with the increase of identiﬁed classes ( i e. , class-incremental learning), jointtraining with all old and new data is too time-consuming and labor-intensive.

Original: Furthermore, mostdata from past tasks are unavailable.
Revised: Furthermore, mostdata from past tasks are unavailable.

Original: A simple alternative is to directly ﬁne-tune the network withnew data.
Revised: A simple alternative is to directly ﬁne-tune the network withnew data.

Original: In this case, the performance of the old class is severely degraded due to the bias of therepresentation and classiﬁer towards the new class, which is also known as catastrophic forgetting.
Revised: In this case, the performance of the old class is severely degraded due to the bias of therepresentation and classiﬁer towards the new class, which is also known as catastrophic forgetting.

Original: To address the issue, existing methods maintain the performance of the old class by preserving therepresentative samples ( i.e. , exemplar) and aligning the output distribution ( i.e. , knowledge distillation).
Revised: To address the issue, existing methods maintain the performance of the old class by preserving therepresentative samples ( i.e. , exemplar) and aligning the output distribution ( i.e. , knowledge distillation).

Original: The joint optimization with incremental samples strikes a balance between stability andplasticity, resulting in better predictions that fall into the label space of the old or new class.
Revised: The joint optimization with incremental samples strikes a balance between stability andplasticity, resulting in better predictions that fall into the label space of the old or new class.

Original: However, is incremental learning always a binary classiﬁcation option that distinguishes between the old and new class?
Revised: However, is incremental learning always a binary classiﬁcation option that distinguishes between the old and new class?

Original: In existing setting ( e.g. , CIFAR100 in Fig.
Revised: In existing setting ( e.g. , CIFAR100 in Fig.

Original: 1), each instance is arranged in a certain incremental stage in an exclusive manner.
Revised: 1), each instance is arranged in a certain incremental stage in an exclusive manner.

Original: However, in real life, people’s semanticunderstanding of the same instance may be gradually enriched as the learning process proceeds.
Revised: However, in real life, people’s semanticunderstanding of the same instance may be gradually enriched as the learning process proceeds.

Original: Asshown in the lower row of Fig.
Revised: Asshown in the lower row of Fig.

Original: 1, the semantics of the same image is reﬁned from vehicles to busin the IIRC-CIFAR setting.
Revised: 1, the semantics of the same image is reﬁned from vehicles to busin the IIRC-CIFAR setting.

Original: This paper focuses on the incremental ability to progressively learn and maintain multi-level semantic information, which is also known as Incremental Implicitly-Reﬁned
Revised: This paper focuses on the incremental ability to progressively learn and maintain multi-level semantic information, which is also known as Incremental Implicitly-Reﬁned

Original: Classiﬁcation ( i.e. IIRC)
Revised: Classiﬁcation ( i.e. IIRC)

Original: How to discern the semantic inheritance relationship in a hierarchical incremental scenery?
Revised: How to discern the semantic inheritance relationship in a hierarchical incremental scenery?

Original: We ﬁnd that the incremental performance for multi-level recognition decreases dramatically whenapplying classical class-incremental learning methods directly to IIRC setting.
Revised: We ﬁnd that the incremental performance for multi-level recognition decreases dramatically whenapplying classical class-incremental learning methods directly to IIRC setting.

Original: Our analysis ismainly attributed to the following reasons.
Revised: Our analysis ismainly attributed to the following reasons.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 2, on the one hand, although theprobability distributions of classes with inheritance relationships show a obvious consistency in theinitial phase, the incremental subclass inherited from a certain old class gradually outgroups underthe supervision of new labels.
Revised: 2, on the one hand, although theprobability distributions of classes with inheritance relationships show a obvious consistency in theinitial phase, the incremental subclass inherited from a certain old class gradually outgroups underthe supervision of new labels.

Original: It destroys the integrity of the representation of the whole old class,thus losing the semantic relevance of the hierarchical labels.
Revised: It destroys the integrity of the representation of the whole old class,thus losing the semantic relevance of the hierarchical labels.

Original: On the other hand, some incrementalclasses inherit from none of the existing classes, leading to the feature confusion due to the lack ofold supervision.
Revised: On the other hand, some incrementalclasses inherit from none of the existing classes, leading to the feature confusion due to the lack ofold supervision.

Original: To this end, we propose an Uncertainty-Aware Hierarchical Reﬁnement (UAHR) scheme, whichexploits the correlation of hierarchical distributions to guide the optimization of incremental representation.
Revised: To this end, we propose an Uncertainty-Aware Hierarchical Reﬁnement (UAHR) scheme, whichexploits the correlation of hierarchical distributions to guide the optimization of incremental representation.

Original: A global representation extension strategy is proposed to widen the distribution distanceamong all new classes in the embedding space, enhancing their discriminative properties.
Revised: A global representation extension strategy is proposed to widen the distribution distanceamong all new classes in the embedding space, enhancing their discriminative properties.

Original: A hierarchical distribution alignment strategy is further proposed to correct the optimization of the shiftingsubclasses by aligning with the distribution of the whole superclass, ensuring the consistency ofthe hierarchical uncertainty.
Revised: A hierarchical distribution alignment strategy is further proposed to correct the optimization of the shiftingsubclasses by aligning with the distribution of the whole superclass, ensuring the consistency ofthe hierarchical uncertainty.

Original: Speciﬁcally, we use RBF mapping to explicitly measure the distancebetween the training samples and the class centers, quantifying the feature correlation.
Revised: Speciﬁcally, we use RBF mapping to explicitly measure the distancebetween the training samples and the class centers, quantifying the feature correlation.

Original: In the incremental phase, we calculate the entropy distribution of new classes in the old embedding spacefor the estimate of hierarchical uncertainty.
Revised: In the incremental phase, we calculate the entropy distribution of new classes in the old embedding spacefor the estimate of hierarchical uncertainty.

Original: The resulting differences and similarities are utilized toidentify the multi-level semantic relationships, reﬁning the subsequent distillation objectives.
Revised: The resulting differences and similarities are utilized toidentify the multi-level semantic relationships, reﬁning the subsequent distillation objectives.

Original: Comprehensive experimental results on IIRC-CIFAR and IIRC-ImageNet-lite datasets demonstrate thesuperiority of our method.
Revised: Comprehensive experimental results on IIRC-CIFAR and IIRC-ImageNet-lite datasets demonstrate thesuperiority of our method.

Original:  Related Work2.1 Uncertainty Estimation
Revised:  Related Work2.1 Class-Incremental Learning

Original: The uncertainty of DNN models [1] mainly comes from the following factors: (1) The diversity inreal-world scenarios.

Original: The inherent error of the measurement system.

Original: The structural error ofthe DNN model.

Original: (4) Errors during DNN training.

Original: (5) Errors caused by unknown data.

Original: For the modeluncertainty, existing solutions can be divided into four types according to the adjustment strategyfor the number of models and model structure.

Original: Single deterministic methods give the predictionbased on one single forward pass within a deterministic network.

Original: J. Van Amersfoort [2] adopts RBFnetwork to learn a linear transformation on the logarithm, and classify the input according to the distance between the transformed logarithm and the class centroids.

Original: In this case, the uncertainty of thedata can be directly estimated from the distance between the centroids.

Original: Bayesian methods cover allkinds of stochastic DNNs.

Original: A simple solution for computing the Gaussian approximation of the lastlayer is proposed [3], and experiments show that the method signiﬁcantly improves the calibration

Original: [Figure]

Original: and alleviates the overconﬁdence prediction of the ReLU network.

Original: Ensemble methods combine thepredictions of several different deterministic networks at inference.

Original: An ensemble training process isintroduced to quantify the prediction uncertainty within the DNN [16], in which member networkis designed with two heads, representing the value and uncertainty of the prediction data.

Original: Test-timeaugmentation methods give the prediction based on one single deterministic network, but augmentthe input data at test-time to generate several predictions, which are used to evaluate the certaintyof the prediction.

Original: A method called "greedy policy search" [17] captures more uncertainty by building a test-time augmentation strategy, which chooses the amount of augmentation to be included ina ﬁxed-length policy.

Original: In this paper, the hierarchical uncertainty is estimated by the class-speciﬁcentropy distribution, promoting the optimization of incremental models.

Original: 2.2 Class-Incremental Learning

Original: Existing class incremental learning methods can be mainly divided into four types.
Revised: Existing class incremental learning methods can be mainly divided into four types.

Original: Regularizationbased methods preserve high-weight parameters by estimating the importance of individual modelparameters [4] [5] [6], while allowing unimportant parameters to be updated ﬂexibly to ensure thelearning ability for new classes.
Revised: Regularizationbased methods preserve high-weight parameters by estimating the importance of individual modelparameters [1] [2] [3], while allowing unimportant parameters to be updated ﬂexibly to ensure thelearning ability for new classes.

Original: However, in the study of [7], it is shown that such methods havepoor generalization performance in class incremental learning.
Revised: However, in the study of [4], it is shown that such methods havepoor generalization performance in class incremental learning.

Original: Distillation-based methods encourage the model to learn new tasks, while the model representation obtained by the current training data is forced to mimic the representation of the old model.
Revised: Distillation-based methods encourage the model to learn new tasks, while the model representation obtained by the current trainingdata is forced to mimic the representation of the old model.

Original: The new model in [8] completesknowledge distillation by matching the output of the sigmoid function with the old model.
Revised: The new model in [5] completes knowledge distillation by matching the output of the sigmoid function with the old model.

Original: LUCIR [9] mitigates forgetting by distilling the output of softmax function with the temperature scaleand L2 regularization.
Revised: LUCIR [6]mitigates forgetting by distilling the output of softmax function with the temperature scale and L2regularization.

Original: LwM [10] reduces forgetting by matching gradient attention maps from thehighest-scoring labels.
Revised: LwM [7] reduces forgetting by matching gradient attention maps from the highestscoring labels.

Original: PODNet [11] performs knowledge distillation between old and new modelsby retaining intermediate features pooled along width and height, and controls the balance betweenprevious knowledge and new information, resulting in a more adaptive incremental representation.
Revised: PODNet [8] performs knowledge distillation between old and new models by retain- [Figure] ing intermediate features pooled along width and height, and controls the balance between previousknowledge and new information, resulting in a more adaptive incremental representation.

Original: Rehearsal-based methods work by storing a limited number of representative examples or adoptinggenerative model for old class samples when training new tasks.
Revised: Rehearsal-based methods work by storing a limited number of representative examples or adoptinggenerative model for old class samples when training new tasks.

Original: Incremental Classiﬁer Representation Learning (iCaRL) [12] retains a small number of samples for each class to approximate classcentroids and make predictions based on the nearest class average classiﬁer.
Revised: Incremental Classiﬁer Representation Learning (iCaRL) [9] retains a small number of samples for each class to approximate classcentroids and make predictions based on the nearest class average classiﬁer.

Original: [13] generates previously observed class samples by using generative adversarial networks (GANS) [14].
Revised: [14] generates previously observed class samples by using generative adversarial networks (GANS) [15].

Original: Structurebased methods keep the learned parameters related to the previous classes unchanged, and assignnew parameters in different forms, such as unused parameters, extra networks, etc., to learn newknowledge.
Revised: Structurebased methods keep the learned parameters related to the previous classes unchanged, and assignnew parameters in different forms, such as unused parameters, extra networks, etc., to learn newknowledge.

Original: DER [15] concatenates the extracted features by adding a new feature extractor in theincremental phase.
Revised: DER [16] concatenates the extracted features by adding a new feature extractor in theincremental phase.

Original: A sparsity error is adopted to encourage the model to compress parameters asmuch as possible.
Revised: A sparsity error is adopted to encourage the model to compress parameters asmuch as possible.

Original: Different from existing incremental method focusing on the invariant semanticconcepts, our method is designed for the progressive understanding of the incremental semanticinformation, facilitating better feature update and retention.My Method
Revised: Different from existing incremental method focusing on the invariant semanticconcepts, our method is designed for the progressive understanding of the incremental semanticinformation, facilitating better feature update and retention.

Original:  [Figure] 3.1 Problem Description and Analysis[Figure]
Revised:  Our Method3.1 Problem Description and Analysis

Original: In real life, people’s semantic understanding of the same thing is gradually enriched as the learning process proceeds.
Revised: In real life, the human brain gradually enriches the semantic understanding of a object as the learningprocess.

Revised: For example, we only knew that rats were small animals when we were young, but weunderstood that rats were rodents as we grew up.

Original: To simulate such a situation, N phases are arranged in order, denoted as{ T 1 , · · · , T N }.Each phase contains three sets, which are used for training, validating, and testing respectively.
Revised: To simulate this process, we adopt the incrementalimplicitly-reﬁned classiﬁcation task (IIRC) as the fundamental task, which is arranges into N phasesin the order of steam data, denoted as { T 1 , · · · , T N }.

Original: Each sample can have one or two labels.
Revised: Each sample can have one or two labels.

Original: In the case of two labels, one label isa subclass and the other label is a superclass.
Revised: Inthe case of two labels, one label is a subclass, and the other label is a superclass.

Original: The number of samples in the superclass is alwayslarger than the subclass, and it increases with the number of subclasses.
Revised: The number ofsamples in the superclass is always larger than the subclass, and it increases with the number ofsubclasses.

Original: During training, we alwaysfollow an incomplete information setup, i.e. , samples with multiple labels are only provided withthe label in the current phase.
Revised: During training, we always follow an incomplete information setup, i.e. , samples withmultiple labels are only provided with the label in the current phase.

Original: In the ﬁrst phase, only a subset of all superclasses is used to train themodel.
Revised: In the ﬁrst phase, only a subsetof all superclasses is used to train the model.

Original: Subsequent phases contain a mixture of superclasses and subclasses.
Revised: Subsequent phases contain a mixture of superclassesand subclasses.

Original: In the training phase,the model is validated in the in-phase validation set which only contains current phase labels.
Revised: In the training phase, the model is validated in the in-phase validation set whichonly contains current phase labels.

Original: In theevaluation phase, the model is evaluated on the post-phase validation set and the test set (both ofthem contain the complete labels).
Revised: In the evaluation phase, the model is evaluated on the post-phasevalidation set and the test set (both of them contain the complete labels).

Original: In this setting, we face two challenging problems: (1) How to judge the relationship between labelsacross phases, especially the superclass-subclass relationship?
Revised: In this setting, we face two challenging problems: (1) How to judge the relationship between labelsacross phases, especially the superclass-subclass relationship?

Original: And how to guide the optimization ofclass distribution to fall into the distribution of superclass exactly?
Revised: And how to guide the optimization ofclass distribution to fall into the distribution of superclass exactly?

Original: (2) How to reduce the confusionamong the feature distributions of new classes in the incremental phase?
Revised: (2) How to reduce the confusionamong the feature distributions of new classes in the incremental phase?

Original: Therefore, We proposetwo strategies.
Revised: Therefore, We proposetwo strategies.

Original: (1) Global Representation Extension.
Revised: (1) Global Representation Extension.

Original: During the new learning phase, the uncertaintybetween each class center is calculated by the RBF kernel.
Revised: During the new learning phase, the uncertaintybetween each class center is calculated by the RBF kernel.

Original: Then the uncertainty is minimized fortraining.
Revised: Then the uncertainty is minimized fortraining.

Original: (2) Hierarchical Distribution Alignment.
Revised: (2) Hierarchical Distribution Alignment.

Original: The uncertainty of the classes across stages arecalculated by calculating the output entropy of the new class samples with old model.
Revised: The uncertainty of the classes across stages arecalculated by calculating the output entropy of the new class samples with old model.

Original: Then it isused to discriminate the superclass-subclass relationship and the brother relationship.
Revised: Then it is used to discriminate the superclass-subclass relationship and the brother relationship.

Original: Finally, theoptimization direction of the new model is guided by changing the output of the old model.
Revised: Finally, theoptimization direction of the new model is guided by changing the output of the old model.

Original: Thus,we can keep the invariance of the superclass distribution.
Revised: Thus,we can keep the invariance of the superclass distribution.

Original: 3.2 Global Representation Extension
Revised: 3.2 Global Representation Extension

Original: In the incremental phase, optimizing only the data of new classes causes great bias in the representations and classiﬁers.
Revised: In the incremental phase, optimizing only the data of new classes causes great bias in the representations and classiﬁers.

Original: To maintain the stability of the feature space, we use the distance betweenrepresentations as a measure of uncertainty.
Revised: To maintain the stability of the feature space, we use the distance betweenrepresentations as a measure of uncertainty.

Original: The optimization of new classes is achieved by comparing the output entropy with the target labels.
Revised: The optimization of new classes is achieved by comparing the output entropy with the target labels.

Original: While the stability of the old class representationsis ensured by maintaining the same output between the old model and the new model with the newsamples.
Revised: While the stability of the old class representationsis ensured by maintaining the same output between the old model and the new model with the newsamples.

Original: Speciﬁcally, we use the RBF distance [2] for the calculation of representation extensionloss.
Revised: Speciﬁcally, we use the RBF distance [13] for the calculation of representation extensionloss.

Original: We use the RBF kernel to map representation to inﬁnite dimensional space for comparison,which is shown as follows,
Revised: We use the RBF kernel to map representation to inﬁnite dimensional space for comparison,which is shown as follows,

Original: [Equation]
Revised: [Equation]

Original: where  n b denotes the number of classes contained in the current batch, h θ ( ∗ ) denotes the outputfeatures of the picture after global pooling, n d denotes the output dimension after global pooling,and h θ ( ∗ ) i denotes the average feature value of a particular class i.
Revised: where K ( ∗ ) denotes the Radial basis function kernel(RBF kernel), n b denotes the number of classescontained in the current batch, h θ ( ∗ ) denotes the output features of the picture after global pooling,n d denotes the output dimension after global pooling, and h θ ( ∗ ) i denotes the average feature valueof a particular class i .

Original: The symbol j in the formula is notequal to c, and j is the class number with the smallest distance to class c in the feature space.
Revised: The symbol j in the formula is not equal to c , and j is the class number withthe smallest distance to class c in the feature space.

Original: Theoptimization of this loss forces the extension of the distance between the class centers, obtainingbetter discrimination in the class distribution.
Revised: The optimization of this loss forces the extensionof the distance between the class centers, obtaining better discrimination in the class distribution.

Original: Subsequently, we use a linear layer to output theentropy of the samples to judge the class of these samples.
Revised: Subsequently, we use a linear layer to output the entropy of the samples to judge the class of thesesamples.

Original: BCEWithLogitsLoss is adopted foroptimization, which is shown as follows,
Revised: BCEWithLogitsLoss is adopted for optimization, which is shown as follows,

Original: [Equation]
Revised: [Equation]

Original: Where Y i denotes the ground truth label of the sample i, N denotes the batch size of training, F idenotes the feature of the i th sample, f θ denotes the classiﬁer of the current model, note that thelabels here as well as the ﬁnal output of the model contain only the classes of current phase.
Revised: Where Y i denotes the ground truth label of the sample i, N denotes the batch size of training, F idenotes the feature of the i th sample, f θ denotes the classiﬁer of the current model, note that thelabels here as well as the ﬁnal output of the model contain only the classes of current phase.

Original: Theoptimization of this loss allows the model to eventually judge the label of a sample by whether itsoutput is greater than 0.
Revised: Theoptimization of this loss allows the model to eventually judge the label of a sample by whether itsoutput is greater than 0.

Original: And the closer to the class center, the larger the output value.
Revised: And the closer to the class center, the larger the output value.

Original: Finally, weuse the output value of the new sample on the old model as a supervised signal.
Revised: Finally, weuse the output value of the new sample on the old model as a supervised signal.

Original: Thus, the currentmodel learns new classes while maintaining the knowledge of the old model.
Revised: Thus, the currentmodel learns new classes while maintaining the knowledge of the old model.

Original: As shown below:
Revised: As shown below:

Original: [Equation]
Revised: [Equation]

Original: Where h θ () denotes the backbone of the current model, f θ () denotes the classiﬁer of the currentmodel, h ∗ θ () denotes the backbone of the old model, f ∗ θ () denotes the classiﬁer head of the oldmodel, α is a scale superparameter of distillation, n old denotes the number of classes that have beenlearned.
Revised: Where h θ () denotes the backbone of the current model, f θ () denotes the classiﬁer of the currentmodel, h ∗ θ () denotes the backbone of the old model, f ∗ θ () denotes the classiﬁer head of the oldmodel, α is a scale superparameter of distillation, n old denotes the number of classes that have beenlearned.

Original: x is a training sample of the current phase.
Revised: x is a training sample of the current phase.

Original: By optimizing this loss, the distribution of oldclass features keeps stable in the feature space.
Revised: By optimizing this loss, the distribution of oldclass features keeps stable in the feature space.

Original: 3.3 Hierarchical Distribution Alignment
Revised: 3.3 Hierarchical Distribution Alignment

Original: In incremental phase, focusing on the label relationships across stages is the key to solving the IIRCproblem.
Revised: In incremental phase, focusing on the label relationships across stages is the key to solving the IIRCproblem.

Original: The old model with ﬁxed parameters can map samples into the old feature space, while thenew subclass samples can be mapped into the space, which is close to the corresponding superclassdistribution.
Revised: The old model with ﬁxed parameters can map samples into the old feature space, while thenew subclass samples can be mapped into the space, which is close to the corresponding superclassdistribution.

Original: We calculate the uncertainty of the labels across phases by counting the entropy of newsamples on old model.
Revised: We calculate the uncertainty of the labels across phases by counting the entropy of newsamples on old model.

Original: Then the label relationships can be inferred from the hierarchical uncertainty.
Revised: Then the label relationships can be inferred from the hierarchical uncertainty.

Original: At the same time, we ﬂexibly adjusting the distribution margin distance, to correct the shift ofsubclass samples and mitigate the confusion caused by the brother class.
Revised: At the same time, we ﬂexibly adjust the distribution border, to correct the shift of subclass samplesand mitigate the confusion caused by the brother class.

Original: Speciﬁcally, as shown in Fig.
Revised: Speciﬁcally, as shown in Fig.

Original: 3, we take the new class C as an example.
Revised: 3, we take the new class C as an example.

Original: We obtain all the mappingfeatures on the old model with the new samples of class C and the output entropy as well.
Revised: We obtain all the mappingfeatures on the old model with the new samples of class C and the output entropy as well.

Original: Thenwe count the single-label output for each phase and sample.
Revised: Thenwe count the single-label output for each phase and sample.

Original: According to the single label matrix,we calculate the uncertainty of a single label, by counting the number of samples, which are judgedto belong to that single class.
Revised: According to the single label matrix,we calculate the uncertainty of a single label, by counting the number of samples, which are judgedto belong to that single class.

Original: And then making the number divide by the number of total samples.
Revised: And then making the number divide by the number of total samples.

Original: Then we process the binarization of uncertainty results.
Revised: Then we process the binarization of uncertainty results.

Original: The label that ﬁnal resultis 1 has asimilar distribution to the current speciﬁc class C (possibly a superclass-subclass or brother relationship).
Revised: The label with ﬁnal result 1 has asimilar distribution to the current speciﬁc class C (possibly a superclass-subclass or brother relationship).

Original: After completing the initial matching of uncertainty distributions, more precise distributionmatching is accomplished by the standard deviation( i.e. , std) values of the global pooling features.
Revised: After completing the initial matching of uncertainty distributions, more precise distributionmatching is accomplished by the standard deviation( i.e. , std) values of the global pooling features.

Original: We calculate the standard deviation of each similar old class with the same operation.
Revised: We calculate the standard deviation of each similar old class with the same operation.

Original: Since directsuperclass-subclass relationships have closer standard deviation values on the feature distributions.
Revised: Since directsuperclass-subclass relationships have closer standard deviation values on the feature distributions.

Original: So the old class label with the closest std value is judged as the superclass of the current class C,and the rest of the classes are brother classes.
Revised: So the old class label with the closest std value is judged as the superclass of the current class C, andthe rest of the classes are brother classes.

Original: The detailed procedure is shown inthe pseudo-code Algorithm 1.
Revised: The detailed procedure is shown in Algorithm 1.

Original: After obtaining the label relationships, we use selective distribution alignment distillation mechanism to guide the representation learning of models.
Revised: After obtaining the label relationships, we use selective distribution alignment distillation mechanism to guide the representation learning of models.

Original: When the new class distribution falls largelywithin the old class, but there is still a partial shift.
Revised: When the new class distribution falls largelywithin the old class, but there is still a partial shift.

Original: We add a margin distance to the output entropyof the corresponding old class.
Revised: We add a margin distance to the output entropyof the corresponding old class.

Revised: The margin distance has a ﬁxed value, as our hyperparameter whencorrecting the inter-hierarchical uncertainty based on output entropy.

Original: When the new class distribution has a crossover interval with theold class, but the gap of standard deviation between the distributions is large, we need to performthe reverse optimization by subtracting a margin distance  value from the output entropy of the oldclass.
Revised: When the new class distribution has a crossover interval with the old class, but the gap of standard deviation between thedistributions is large, we need to perform the reverse optimization by subtracting a margin distance [Table] value from the output entropy of the old class.

Original: So we can achieved the alignment of the distribution and mitigate the confusion caused by thebrother class by this strategy.
Revised: So we can achieved the alignment of the distributionand mitigate the confusion caused by the brother class by this strategy.

Original: Speciﬁcally, according to the label relationships, when the current new class C is not related to theold class, the highest output entropy by the sample of class C on the old model, is subtracted by amargin value.
Revised: Speciﬁcally, according to the label relationships, when the current new class C is not related to theold class, the highest output entropy by the sample of class C on the old model, is subtracted by amargin value.

Original: And when the current new class C is a subclass of the old class, the output entropyvalue of the class C sample on the old model, which is corresponding to the superclass, is added onemargin value.
Revised: And when the current new class C is a subclass of the old class, the output entropyvalue of the class C sample on the old model, which is corresponding to the superclass, is added onemargin value.

Original: At the same time, the highest output entropy value of the non-superclass is subtractedby a margin value.
Revised: At the same time, the highest output entropy value of the non-superclass is subtractedby a margin value.

Original: Our selective distribution alignment distillation loss can be obtained by:
Revised: Our selective distribution alignment distillation loss can be obtained by:

Original: [Equation]
Revised: [Equation]

Original: where, y new is the new output of the old model, after performing our hierarchical distribution alignment strategy.
Revised: where y new is the new output of the old model, after performing our hierarchical distribution alignment strategy.

Original: In the end, our complete loss is:
Revised: In the end, our complete loss is:

Original: [Equation]
Revised: [Equation]

Original: where γ is the learning weight for the optimization of the global representation extension.
Revised: where γ is the learning weight for the optimization of the global representation extension.

Revised: l cls andl dis represents the same BCEwithLogitsLoss, in which the outputs are of the same magnitude, weset their ratio to 1:1 for close loss values.

Revised: And in our experiments, γ is set as 10.0.

Revised: A detaileddescription of the hyperparameter selection is shown in the supplementary material B.2.

Original:  Experiments
Revised:  Experiments

Original: 4.1 Dataset and Settings
Revised: 4.1 Dataset and Settings

Original:  Based on the semantic relevance of labels, CIFAR100 [19] and ImageNet [20] datasets are integratedto form datasets, where class labels are two-level hierarchy[18].
Revised: [Figure] Based on the semantic relevance of labels, CIFAR100 [18] and ImageNet [19] datasets are integratedto form datasets, where class labels are two-level hierarchy[17].

Original: Each label starts as a leaf node (i.e.,subclass) and similar labels are assigned a common parent node (i.e., superclass).
Revised: Each label starts as a leaf node (i.e.,subclass) and similar labels are assigned a common parent node (i.e., superclass).

Original: The integrateddatasets are called IIRC-CIFAR and IIRC-ImageNet-lite.
Revised: The integrateddatasets are called IIRC-CIFAR and IIRC-ImageNet-full.

Original: IIRC-CIFAR.
Revised: IIRC-CIFAR.

Original: Ten superclasses are setup, each with about 4 to 8 subclasses.
Revised: Ten superclasses are setup, each with about 4 to 8 subclasses.

Original: In incremental phases, each new phase introduces 5 classesfor IIRC-CIFAR.
Revised: In incremental phases, each new phase introduces 5 classesfor IIRC-CIFAR.

Original: In total, IIRC-CIFAR involves 22 phases with 10 preset class orders for multipletests, which is called as phase conﬁguration.
Revised: In total, IIRC-CIFAR involves 22 phases with 10 preset class orders for multipletests, which is called as phase conﬁguration.

Original: IIRC-ImageNet-lite.
Revised: IIRC-ImageNet-full.

Original: In IIRC-ImageNet-lite, sixtythree superclasses are set up, and the number of subclasses that belong to one superclasses variesgreatly, from 3 to 118.
Revised: In IIRC-ImageNet-full, sixtythree superclasses are set up, and the number of subclasses that belong to one superclasses variesgreatly, from 3 to 118.

Original: There are a total of 10 phases with 30 classes per phase.
Revised: There are a total of 35 phases with 30 classes per phase.

Original: 5 preset class ordersare ﬁxed for multiple tests.
Revised: 5 preset class ordersare ﬁxed for multiple tests.

Revised: IIRC-ImageNet-lite is a shorter, lighter version which has just ten tasks(with ﬁve tasks conﬁgurations), while referring to both collectively as IIRC-ImageNet-full.

Revised: IIRCImageNet-Subset [11], as a simpliﬁed version of the IIRC-ImageNet-full, has 10 superclasses and100 subclasses (including those have no superclass labels).

Original: The superclasses of each dataset are combined by extracting 40% samples from each subclass, whilethe corresponding subclasses are saved with 80% samples, so the superclasses and subclasses share [Table] [Table] 20% samples.
Revised: The superclasses of each dataset are combined by extracting 40% samples from each subclass, whilethe corresponding subclasses are saved with 80% samples, so the superclasses and subclasses share20% samples.

Original: When the superclass contains more than 8 subclasses, the samples of the subclassesare extracted in the proportion of
Revised: When the superclass contains more than 8 subclasses, the samples of the subclassesare extracted in the proportion of

Original: [Table]
Revised: ∗ 40% .

Original: The training process follows theincomplete information setting, i.e. , if a sample has more than one label, only the label of the currentphase is provided.
Revised: The training process follows theincomplete information setting, i.e. , if a sample has more than one label, only the label of the currentphase is provided.

Original: The validation set also follows the incomplete information setting.
Revised: The validation set also follows the incomplete information setting.

Original: After a certainnumber of training phases, the overall performance is evaluated on a validation set with completeinformation setting and a test set as well.
Revised: After a certainnumber of training phases, the overall performance is evaluated on a validation set with completeinformation setting and a test set as well.

Original: Both of them contain samples of all seen classes and theircorresponding complete labels.
Revised: Both of them contain samples of all seen classes and theircorresponding complete labels.

Revised: [Table]

Original: Metrics.
Revised: Metrics.

Original: The evaluation metric uses the precision-weighted Jaccard similarity (PW-JS) to measurethe performance of the model on phase k after training on phase j, as follows:
Revised: The evaluation metric uses the precision-weighted Jaccard similarity (PW-JS) to measurethe performance of the model on phase k after training on phase j, as follows:

Original: [Equation]
Revised: [Equation]

Original: In the formula j ≥ k , b Y ki is the prediction value of the model for i th samples of k th phase, Y ki is theground truth, and n k is the number of samples in this phase.
Revised: In the formula j ≥ k , b Y ki is the prediction value of the model for i th samples of k th phase, Y ki is theground truth, and n k is the number of samples in this phase.

Original: And for the evaluation of the overallperformance, we use the average precision-weighted Jaccard similarity of the model after trainingon phase j for all seen classes, as follows.
Revised: And for the evaluation of the overallperformance, we use the average precision-weighted Jaccard similarity of the model after trainingon phase j for all seen classes, as follows.

Original: [Equation]
Revised: [Equation]

Original: 4.2 Ablation Study
Revised: 4.2 Ablation Study

Original: To prove the effectiveness of our method, we conduct a set of ablation experiments on the IIRCCIFAR dataset.
Revised: To prove the effectiveness of our method, we conduct a set of ablation experiments on the IIRCCIFAR dataset.

Original: One of the important components in our scheme is the global representation extension strategy, referred to as the GRE component.
Revised: One of the important components in our scheme is the global representation extension strategy, referred to as the GRE component.

Original: The other is hierarchical distribution alignmentstrategy, referred to as the HDA component.
Revised: The other is hierarchical distribution alignmentstrategy, referred to as the HDA component.

Original: To verify the functionality of the GRE and HDA components, we conducted experiments in the case of phase conﬁguration 0 of IIRC-CIFAR.
Revised: To verify the functionality of the GRE and HDA components, we conducted experiments in the case of phase conﬁguration 0 of IIRC-CIFAR.

Original: As shownin Table 1, the HDA component alone achieves a improvement of 0.6% in the last phase, while the GRE component keeps the same results as baseline with an initial lag of 2.68%.
Revised: As shownin Table 1, the HDA component alone achieves a improvement of 0.6% in the last phase, whilethe GRE component alone achieve a 0.54% in the last phase.

Original: And with bothcomponents added, 1.27% increase over baseline is achieved.
Revised: And with both components added,1.27% increase over baseline is achieved.

Original: It proves that both components play apositive role in the evaluation of the overall incremental performance.
Revised: It proves that both components play a positive role in theevaluation of the overall incremental performance.

Original: In Fig.
Revised: In Fig.

Original: 5 (C), we show the performance of each phase, which is evaluated over the test samples ata speciﬁc phase j, after training on that phase( R jj using Equation 6).
Revised: 5 (C), we show the performance of each phase, which is evaluated over the test samples ata speciﬁc phase j, after training on that phase( R jj using Equation 6).

Original: Such cure graphs evaluatethe learning ability for new classes.
Revised: Such cure graphs evaluatethe learning ability for new classes.

Original: We can see that the HDA component alone achieves a greatimprovement, while adding the GRE component further enhances the learning ability with a smalloptimization on top of the HDA component. 
Revised: We can see that the HDA component alone achieves a greatimprovement, while adding the GRE component further enhances the learning ability with a smalloptimization on top of the HDA component. [Figure]

Original: 4.3 Analysis
Revised: 4.3 Analysis

Original: The impact of global representation extension.
Revised: The impact of global representation extension.

Original: To explore the impact of global representationextension strategy, we visualize the t-SNE features of the baseline and our method.
Revised: To explore the impact of global representationextension strategy, we visualize the t-SNE features of the baseline and our method.

Original: We select data [Figure] from phase 0 and phase 4 of IIRC-CIFAR with the phase conﬁguration 0.
Revised: We select datafrom phase 0 and phase 4 of IIRC-CIFAR with the phase conﬁguration 0.

Original: In the upper part of Fig.4, we can see that there is an overlap among the representation distributions of the new classes inthe baseline method.
Revised: In the upper part of Fig.4, we can see that there is an overlap among the representation distributions of the new classes inthe baseline method.

Original: While the overlap between the increasing classes is weakened and the distancebetween the increasing class feature distributions is expanded in our method.
Revised: While the overlap between the increasing classes is weakened and the distancebetween the increasing class feature distributions is expanded in our method.

Original: This proves that usingthe distance between representations as a measure of uncertainty is effective and the reduction ofthis uncertainty leads to the extension of the distance between the individual class centers.
Revised: This proves that usingthe distance between representations as a measure of uncertainty is effective and the reduction ofthis uncertainty leads to the extension of the distance between the individual class centers.

Original: The impact of the hierarchical uncertainty.
Revised: The impact of the hierarchical uncertainty.

Original: To explore the effect of hierarchical uncertainty, weﬁrst compare different uncertainty methods.
Revised: To explore the effect of hierarchical uncertainty, weﬁrst compare different uncertainty methods.

Revised: A detailed version of Table 2 is presented in A.3 of

Revised: Supplementary Materials.

Original: As shown in Table.
Revised: As shown in Table.

Original: 2, we use other common values toconstruct the label relationships.
Revised: 2, the "Common Uncertainty" denotes an entropy- [Figure] based uncertainty method which is widely used in the OOD [20]. We use it to construct the labelrelationships.

Original: It can be seen from the Table.
Revised: It can be seen from the Table.

Original: 2 that our method achieves betterexperimental results, which proves that the entropy output of the network and the standard deviation[Figure] of the features have the greatest impact on the cross-hierarchical uncertainty estimation among thestatistical properties of the superclass and the corresponding subclass.
Revised: 2 that our method achieves better experimental results,which proves that the entropy output of the network and the standard deviation of the features havethe greatest impact on the cross-hierarchical uncertainty estimation among the statistical propertiesof the superclass and the corresponding subclass.

Original: To explore the effect of the hierarchical distribution alignment strategy.
Revised: To explore the effect of the hierarchical distribution alignment strategy.

Original: We visualize the t-SNE features for the baselinemethod and our method onthe IIRC-CIFAR dataset with phase conﬁguration0 by selecting data from phase 0 and phase 11.
Revised: We visualize the t-SNEfeatures for the baseline and our method on IIRC-CIFAR dataset with phase conﬁguration 0 by selecting data from phase 0 and phase 11.

Original: In the bottom half of Fig. 4, we can see that thereis a large shift between the distributions of subclasses and corresponding superclasses in baseline.
Revised: In the bottom half of Fig. 4, we can see that there is a largeshift between the distributions of subclasses and corresponding superclasses in baseline.

Original: While this shift is greatly corrected in our method,so that the distribution of the increasing subclasses falls in the distribution of the corresponding superclasses.
Revised: While thisshift is greatly corrected in our method, the distribution of the increasing subclasses falls in the distribution of the corresponding superclasses.

Original: It demonstrates that adjusting theoptimization direction about the feature distribution according to the labeling relationship guidesthe new model to better learn the representational relationship between the old and new classes anddeepen it gradually.
Revised: It demonstrates that adjusting the optimization directionabout the feature distribution according to the labeling relationship guides the new model to betterlearn the representational relationship between the old and new classes and deepen it gradually.

Original: 4.4 Comparison with SOTA
Revised: 4.4 Comparison with SOTA

Original: To comprehensively measure the performance of our method under IIRC setting, we reproduced theclassical methods ER, AGEM, LUCIR, iCaRL-CNN, iCaRL-norm, podnet, etc and compare withthem.
Revised: To comprehensively measure the performance of our method under IIRC setting, we reproduces theclassical methods ER [10], AGEM [12], LUCIR [6], iCaRL-CNN [9], iCaRL-norm [9], Podnet [8], HCV [11] etc and compare with them.

Original: Average PW-JS.
Revised: Average PW-JS.

Original: It can be seen in Fig.
Revised: It can be seen in Fig.

Original: 5 (A) that our method achieves the best until the end amongall methods with the incomplete information in IIRC-CIFAR.
Revised: 5 (A) that our method achieves the best until the end amongall methods with the incomplete information in IIRC-CIFAR.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 5 (D)
Revised: 5 (D)

Original: , our methodsalso maintains the highest scores in larger IIRC-imagenet-lite, which demonstrates the effectivenessof our methods.
Revised: , our methodsalso maintains the highest scores in larger IIRC-imagenet-lite, which demonstrates the effectivenessof our methods.

Revised: As shown in Fig.

Revised: 5 (E), we achieve the SOTA effect on the IIRC-ImageNet-Subsetdataset.

Revised: Fig.

Revised: 5(F) shows that we ﬁnally achieved the highest score on IIRC-ImageNet-full.

Revised: Moredetails are shown in the A.5 of supplementary materials.

Original: Performance on new classes.
Revised: Performance on new classes.

Original: The PW-JS values indicating the performance of newly incrementalphase are shown in Fig.
Revised: The PW-JS values indicating the performance of newly incrementalphase are shown in Fig.

Revised: 5(B).

Original: It can be seen that the ER algorithm and the AGM algorithm are superiorin the ability to learn new classes.
Revised: It can be seen that the ER algorithm and the AGM algorithm aresuperior in the ability to learn new classes.

Original: Combined with their performance on overall phases, it is clearthat they obtain an advantage on the new class at the complete expense of the old class performance.
Revised: Combined with their performance on overall phases, itis clear that they obtain an advantage on the new class at the complete expense of the old class performance.

Original: Among all algorithms that are beneﬁcial for suppressing the catastrophic forgetting, our methodachieves the best, and outperforms existing methods in the ability of learning new classes.
Revised: Among all algorithms that are beneﬁcial for suppressing the catastrophic forgetting, ourmethod achieves the best, and outperforms existing methods in the ability of learning new classes.

Original: We conduct an additional set of confusion graph experiments where we extracted the data from the IIRC-CIFAR with the phases 0, 1, 2, 7, 12, 17.
Revised: We conduct an additional set of confusion graph experiments where we extracted the data fromthe IIRC-CIFAR with the phase 0, 1, 2, 7, 12, 17.

Original: They are combined and presented in Fig.
Revised: They are combined and presented in Fig.

Original: We canﬁnd that the LUCIR algorithm has a strong retention effect for old classes, while it can learn part ofthe labeling relationship when the number of phases is relatively small, but presents a weakness inlearning new classes of phases when the number of phases increases. The iCaRL-CNN algorithm,on the other hand, is able to maintain old classes and learn new classes, but is very confusing forthe construction of labeling relations.
Revised: In the IIRC task, the diagonal values in the confusion matrix represents the prediction of subclasslabels, while the bottom-left corner represents the one of superclasses. A more accurate confusionmatrix in the lower left corner represents better label relationship maintenance between subclassand superclass.

Original: Our approach is able to maintain old classes and learn newclasses more accurately, making a great improvement in maintaining label relations.
Revised: As shown in Figure 6, our method performs best in both the diagonal and lower leftpositions, proving the superiority of our approach in maintaining label relationships.

Original:  Conclusion
Revised:  Conclusion

Original: In this paper, a novel Uncertainty-Aware Hierarchical Reﬁnement scheme is proposed for IIRC task.
Revised: In this paper, a novel Uncertainty-Aware Hierarchical Reﬁnement scheme is proposed for IIRC task.

Original: A global representation extension strategy is presented to enhance the discrimination of incrementalclasses, and the misleading distillation process is reﬁned with a hierarchical distribution alignmentstrategy.
Revised: A global representation extension strategy is presented to enhance the discrimination of incrementalclasses, and the misleading distillation process is reﬁned with a hierarchical distribution alignmentstrategy.

Original: Consequently, our method involves a multi-level semantic scenery into the incrementallearning.
Revised: Consequently, our method involves a multi-level semantic scenery into the incrementallearning.

Original: Experimental results show the superiority our method in both stability and plasticity.
Revised: Experimental results show the superiority our method in both stability and plasticity.

Original: Checklist
Revised: Checklist

Original: The checklist follows the references.
Revised: The checklist follows the references.

Original: Please read the checklist guidelines carefully for informationon how to answer these questions.
Revised: Please read the checklist guidelines carefully for informationon how to answer these questions.

Original: For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .
Revised: For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .

Original: You are strongly encouraged to include a justiﬁcation to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
Revised: You are strongly encouraged to include a justiﬁcation to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.

Original: For example:
Revised: For example:

Original: • Did you include the license to the code and datasets?
Revised: • Did you include the license to the code and datasets?

Original: [Yes] See Section 2.•
Revised: [Yes] See Section 2.•

Original: Did you include the license to the code and datasets?
Revised: Did you include the license to the code and datasets?

Original: [No] The code and the data areproprietary.•
Revised: [No] The code and the data areproprietary.•

Original: Did you include the license to the code and datasets?
Revised: Did you include the license to the code and datasets?

Original: [N/A]
Revised: [N/A]

Original: Please do not modify the questions and only use the provided macros for your answers.
Revised: Please do not modify the questions and only use the provided macros for your answers.

Original: Note that the
Revised: Note that the

Original: Checklist section does not count towards the page limit.
Revised: Checklist section does not count towards the page limit.

Original: In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
Revised: In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.

Original: For all authors...
Revised: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s
Revised: (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s

Original: contributions and scope?
Revised: contributions and scope?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you describe the limitations of your work?
Revised: (b) Did you describe the limitations of your work?

Original: [Yes] See Supplementary Materials.(c)
Revised: [Yes] See Supplementary Materials.

Original:  Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [Yes] SeeSupplementary Materials.
Revised: [Yes] SeeSupplementary Materials.

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes]2.
Revised: [Yes]

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A](b)
Revised: [N/A]

Original:  Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [N/A]3.
Revised: [N/A]

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material or as a URL)?

Original: [Yes] See Supplementary Materials.
Revised: [Yes] See Supplementary Materials.

Original: (b) Did you specify all the training details (e g., data splits, hyperparameters,
Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters,

Original: how they
Revised: how they

Original: were chosen)?
Revised: were chosen)?

Original: [Yes]
Revised: [Yes]

Original: (c) Did you report error bars (e.g., with respect to the random seed after running experi-
Revised: (c) Did you report error bars (e.g., with respect to the random seed after running experi-

Original: ments multiple times)?
Revised: ments multiple times)?

Original: [Yes]
Revised: [Yes]

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?

Original: [Yes] See Supplementary Materials.
Revised: [Yes] See Supplementary Materials.

Original: If you are using existing assets (e
Revised: If you are using existing assets (e

Original: g., code, data, models) or curating/releasing new assets...
Revised: g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [Yes]
Revised: [Yes]

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [N/A]
Revised: [N/A]

Original: (d) Did you discuss whether and how consent was obtained from people whose data
Revised: (d) Did you discuss whether and how consent was obtained from people whose data

Original: you’re using/curating?
Revised: you’re using/curating?

Original: [N/A]
Revised: [N/A]

Original: (e) Did you discuss whether the data you are using/curating contains personally identiﬁ-
Revised: (e) Did you discuss whether the data you are using/curating contains personally identiﬁ-

Original: able information or offensive content?
Revised: able information or offensive content?

Original: [N/A]
Revised: [N/A]

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...

Original: (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
Revised: (a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you describe any potential participant risks, with links to Institutional Review
Revised: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: Board (IRB) approvals, if applicable?
Revised: Board (IRB) approvals, if applicable?

Original: [N/A]
Revised: [N/A]

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount
Revised: (c) Did you include the estimated hourly wage paid to participants and the total amount

Original: spent on participant compensation?
Revised: spent on participant compensation?

Original: [N/A]
Revised: [N/A]