Grow and Merge: A Unified Framework for Continuous Categories Discovery
Anonymous Author(s)
Abstract
Although a number of studies are devoted to novel category discovery, most ofthem assume a static setting where both labeled and unlabeled data are given atonce for finding new categories.
In this work, we focus on the application scenarioswhere unlabeled data are continuously fed into the category discovery system.
We refer to it as the Continuous Category Discovery ( CCD ) problem, whichis significantly more challenging than the static setting.
A common challengefaced by novel category discovery is that different sets of features are needed forclassification and category discovery: class discriminative features are preferred forclassification, while rich and diverse features are more suitable for new categorymining.
This challenge becomes more severe for dynamic setting as the systemis asked to deliver good performance for known classes over time, and at thesame time continuously discover new classes from unlabeled data.
To addressthis challenge, we develop a framework of Grow and Merge ( GM ) that works byalternating between a growing phase and a merging phase: in the growing phase, itincreases the diversity of features through a continuous self-supervised learning foreffective category mining, and in the merging phase, it merges the grown modelwith a static one to ensure satisfying performance for known classes.
Our extensivestudies verify that the proposed GM framework is significantly more effective thanthe state-of-the-art approaches for continuous category discovery.
Introduction
Human beings are good at grouping objects into category through clustering, and the definition ofcategories are continuously expanding and updated over time.
Recent developments of intelligentvisual systems can not only distinguish pre-defined categories [1–5], but also discover new categoriesfrom unlabeled data, a task that is known as novel category discovery [6–8].
Existing works on novel category discovery are limited to the static setting, where both labeleddata (by known classes) and unlabeled data (with potential unknown categories) are given at once.
In contrast, for real-world applications, unlabeled data are continuously fed into the system fordiscovering new categories, making it a significantly more challenging problem.
Besides, currentstudies for novel category discovery often assume that all the unlabeled data belong to the unknownnew categories, which is generally not true in real applications.
In this work, we examine the dynamicsetup of novel category discovery where the system was initially given a set of data labeled by knownclasses, and unlabeled data are continuously streamed into the system for discovering new classes.
The system is requested to consistently yield satisfying performance for known classes, and at thesame time, dynamically discover new categories from the streaming unlabeled data.
We refer to it as
Continuous Category Discovery , or CCD for short.
We illustrate the process of CCD in Figure 1.
It is comprised of two main stages: the initial stagewhere a classification model is trained by a set of labeled examples, and the continuous categorydiscovery stage where new categories are continuously discovered from a stream of unlabeleddata belonging to both known and unknown classes.
A intuitive approach to address the dynamicnature of CCD is to combine the existing methods for open-set recognition [9, 10], novel categorydiscovery [11, 6, 12], and incremental learning [13, 14].
This is however insufficient because ourlearning system has to accomplish two tasks at the same time, i.e. , accurately classify instances intothe known classes, and discover new categories from an unlabeled data stream.
It turns out that thesetwo task models usually produce different types of features: discriminative features on known classesare preferred by classification model, while rich and diverse features are critical for identifying newclasses, as illustrated in Figure 2.
A simple combination of novel category discovery and incrementallearning will fail to address the trade-off consistently over time, which is further verified by ourempirical studies.
[Figure]
To address the challenge of continuous category discovery, we propose a framework of Grow and
Merge , or GM for short.
After pre-training a static model A over the labeled data, we will updatemodel A with respect to unlabeled data stream by alternating between the growing phase and themerging phase: in the growing phase, we will increase the diversity of features by continuouslytraining our model over received unlabeled data through a combination of supervised and selfsupervised learning; in the merging phase, we will merge the grown model with the static one bytaking a weighted combination of both models.
By alternating between the growing and mergingphases, we are able to maintain a good performance for known classes, and at same time, the powerof discovering new categories.
This is clearly visualized in Figure 2, where the first two panels showthat existing approaches can do well on one of the two tasks but not both, and last panel shows thatfeatures learned by the proposed GM framework works well for both tasks.
Finally, one of the common issues with continuous training is catastrophic forgetting.
To alleviatingthe forgetting effect as we are growing the number of categories over time, we maintain a smallset of labeled samples from known categories and pseudo-labeled samples from novel categories.
These selected examples are used in the growing phase to expand feature diversity for effectivecategory discovery.
Extensive experimental results show that our proposed method consistentlyshows satisfying performance under multiple practical scenarios compared with existing methods.
The main contributions of this paper are summarized as follows:
• We study a new problem named continuous category discovery, or CCD, which better
reflects the challenge of category discovery in the wild.
It needs to simultaneously maintaina good performance for known categories and the ability of discovering novel categories.
• We propose a framework of grow and merge, or GM, for CCD, that is able to resolve the
conflicts between the classification task and the task of discovering new categories.
• We conduct experiments under four different settings to fully investigate the scenarios ofnovel category discovery in the wild.
The proposed method shows less forgetting of knowncategories and better performance for category discovery compared to existing methods.
Problem Definition
In this section, we formulate the problem of Continuous Category Discovery (CCD).
There are twomain stages of the CCD problem, i.e. , Initial Stage and Continuous Category Discovery Stage .
The settings of each stage are introduced, together with the evaluation metrics of CCD problem.
2.1 Setting of Continuous Category Discovery
During the initial stage, a labeled training dataset D 0 train = { ( x 0 i , y 0 i ) } N 0 i =1 is provided to train themodel on the initial known category set C
= { 1 , 2 , ..., K 0 } , where x 0 i is the initial training data, andy 0 i ∈
C 0 is the corresponding label.
The model is expected to classify categories in C 0 , and learnmeaningful representation from the labels’ semantic information.
During the continuous category discovery stage, a serial of unlabeled training datasets {D ttrain } Tt =1are sequentially provided, where D ttrain = { x ti } N t
i
=1 indicates the dataset at time t .
Though unlabeled,we denote the known or potential appeared categories until time t as C t = { 1 , 2 , ..., K t } .
The modelis expected to discover newly appeared unknown categories from D ttrain , store the representations ofthe discovered novel categories and maintain the knowledge of the known categories.
2.2 Evaluation Metrics
At each time-step t , we evaluate the classification performance on the test dataset D ttest ={ ( x si , y si ) | s ≤ t } , containing test samples from all the known or previously discovered categories.
Forthe newly appeared unknown categories, we evaluate the novel category discovery performance.
Themaximum forgetting metric M f and the final discovery metric M d are designed for evaluation.
To evaluate the performance of the clustering assignments, we follow the standard practise [6, 15] toadopt clustering accuracy on the known categories and the novel categories, denoted as ACC t knownand ACC t novel , respectively.
The maximum forgetting M f is defined as the maximum value of thedifferences between ACC 0 known and ACC t known for every t and the final discovery M d is definedas the final cluster accuracy on novel categories, i.e. , M d = ACC T novel .
See more details in thesupplementary materials.
Method
3.1 An Overview of Grow and Merge Framework
The CCD problem requires both maintaining the performance on known categories and discoveringnovel categories under the continuous learning scheme.
For the compatibility of the two tasks, wepropose a Grow and Merge (GM) Framework, which is illustrated in Figure 3.
During the initial stage, the model ϕ 0 ( · ) is firstly pre-trained on the initial dataset with labeled datain both supervised and self-supervised manner.
An exemplars set P k
= { p k,i } is constructed as
[Figure]
well as the prototypes .
The exemplar set contains representative samples for each class while theprototype µ k is the averaged feature vector of the exemplars from the k -th class.
The exemplar set isconstructed by iteratively picking several typical samples with the principle of maintaining the closestcenter to the prototype µ k , i.e. p k,i ← argmin x ∥ µ k − 1 i [ ϕ 0 ( x ) + Σ
i − 1 j =1 ϕ
( p k , j )] ∥ .
The definitionsof prototype and exemplars are adapted from iCaRL [14], which enables us to use a non-parametricclassifier that predicts label of a query sample from its nearest prototype’s label.
During the continuous category discovery stage, a dual-branch architecture with a static branch ϕ S ( · )and a dynamic branch ϕ D ( · ) is introduced for training, where both branches are initialized by thepre-trained model ϕ 0 ( · ) .
At each time-step, GM includes two phases: Growing and Merging .
In thegrowing phase, the novel category data are filtered out from the continuous data and the dynamicbranch is trained for novel category discovery.
Then the merging phase fuses the newly discoveredand known categories into a single model for unified classification.
Knowledge from the frozen staticbranch is distillated into the dynamic one to avoid forgetting previous classes.
It is noted that though GM maintains two branches during training, only one branch is needed duringtesting.
Experimental results show significant performance of the GM framework, where the dual-brancharchitecture and knowledge distillation succeed to balance the feature conflicts between continuousclassification and novel categories discovery, and show better performance on both sub-tasks.
3.2 Growing: Enrich Features with Unlabeled Data
To achieve novel category discovery from the data stream with both labeled and unlabeled samplesfrom known and unknown classes, we firstly filter out data of unknown classes with novelty detection ,and then conduct novel category discovery based on both the new data and the previous model.
Novelty Detection aims to identify the samples belonging to the novel categories in the currenttraining dataset D ttrain , since there may also exist samples from previously known categories.
With the maintained prototypes of known categories, the feature vectors of samples from knowncategories would lay closer to at least one prototype with a high probability.
Then a novel distanced novel is defined as the distance between the sample x ti and its nearest prototype, i.e. d novel ( x ti ) =
min 1 ≤ k ≤ K t d ( µ k , ϕ tD ( x ti )) , where d ( · , · ) is the distance function.
Samples with d novel larger than athreshold ϵ are filtered out as novel samples for the following process.
Novel Categories Discovery and Learning from Static Branch .
We optimize the dynamic branchϕ tD ( · ) for novel category discovery, guided by both the novel samples and previously learnedknowledge for rich feature representation.
(i) For learning and discovering from the novel samples,we adopt an unsupervised representation learning method [6].
A cluster head g ( · ) is learned to assignsamples ˆ x ti to the clusters with novel category indices { K t − 1 + 1 , ..., K t } .
Such an assignment is supervised by the similarity s ij between samples ˆ x ti and ˆ x tj with a binary cross entropy loss, i.e.
[Equation]
where ˆ N t is the number of novel category samples at time t (specified or estimated).
The similaritys ij is measured by the Winner-Take-All hash (WTA), which equals one when samples share the sametop- k channels of their feature vector, otherwise equals zero.(ii)
For learning from the static branch ϕ tD ( · ) with previously learned knowledge, we introduce astatic-dynamic distillation loss to transfer the representation learned from labeled data in the staticbranch into the dynamic one:
[Equation]
The joint loss L BCE and L SD enables the dynamic branch to discovering novel categories with richerfeature representation, and the cluster head g ( · ) could assign new labels to the novel category data ˆ x ti .
3.3 Merging: Unification of Categories and Branches
After the Growing phase, both categories and models contain two parts for the known and novelcategories, respectively.
The Merging phase reunifies them together for all the appeared categories.
3.3.1 Category Unification: Continuous Learning for Newly Discovered Categories
To merge the novel categories discovered in Growing phase into the classifier, we firstly sift samplesto construct the exemplar sets of the novel categories, then refine the model representation using apseudo label representation learning for classification on both known and novel classes.
Sample Sifting aims to filter out incorrectly-assigned samples based on their cluster confidence.
Theassigned class indices by g ( · ) are denoted as the pseudo-labels of the novel samples, which mayinevitably have errors and lead to less reliable prototypes of the novel categories.
We sift out thesepotentially noisy samples via the local sample density, with the assumption that samples from thesame class would lay tightly.
Suppose N j
(ˆ z ti ) indicates the j nearest neighbors of ˆ z ti from the sameclass.
The local sample density G j (ˆ z ti ) is defined as the distance between ˆ z ti and its j -th nearestneighbors, i.e. G j (ˆ z ti ) =
max z ∈N j (ˆ z ti ) d
(ˆ z ti , z ) .
Samples with larger G j are sifted out, then the leftsamples are denoted as ˜ x
ti with pseudo labels ˜
c ti .
Pseudo Label Representation Learning.
Based on the sifted novel samples of the k -th novelcategory, we firstly construct the exemplar set P k
= { p k,i } .
Without knowing the prototype µ k , i.e.the explicit center of the k -th category, we estimate it with the averaged vectors of sifted samples.
For the purpose of continuous classification on all appeared classes, it is required to tighten thesparse representation distribution learned in the growing phase.
Pseudo Label Learning (PLL) loss isintroduced here.
Compared with commonly-used Cross Entropy loss, PLL does not require an explicitclassifier and thus does not modify the deployed model structure during the continuous phase.
Theloss function L PLL aims to pull exemplars pull the exemplars p k and the corresponding prototypesµ k closer, and push p k away from the other prototypes µ j ( j ̸ = k ) :
[Equation]
(3) where τ is a temperature hyper-parameter.
To avoid catastrophic forgetting, based on data in the latestexemplars, the static-dynamic distillation loss in Eq.
2 is also employed here.
3.3.2 Branch Unification: Merging Multiple Branches into One Unified Model
During the above process, the dynamic branch is optimized to classify all appeared categories.
Inorder to reduce computation cost and further utilize the representation of the static branch trainedfrom labeled data, we merge the two branches into a single one.
In experiments, we found such a branch merging can further improve the recognition performance.
To fuse the representation of ϕ Sand ϕ tD , the exponential moving average strategy (EMA) is applied:
[Equation]
where α is the momentum hyper-parameter to control the weight of fusion, generally valued close to1 ( α = 0 .
99 in this paper).
Experiments
In this section, we design experiments in different scenarios to simulate the varying cases of real-worldapplications, which enables us to fully investigate the performance and flexibility of the proposedmethod.
Two main evaluation metrics, namely, maximum forgetting M f and final discovery M d areemployed to measure the performance.
CIFAR-100 [16], CUB-200 [17] and ImageNet-100 [18, 19]representing the small, middle and large scale datasets are used in the experiments.
Experimental Scenarios .We formulate four different scenarios for experiments as follows.
(1) Class
Incremental Scenario (CI) : the data are only drawn from novel categories.
(2) Data Incremental Scenario (DI) : the data are only drawn from the known categories.
(3) Mixed Incremental Scenario (MI) :the data are drawn from both novel and known categories.
(4) Semi-supervised Mixed Incremental
Scenario (SMI) : the data are drawn from both novel and known categories, and a portion of the dataare labeled, which is closed to the real-world application.
[Table]
The settings of these scenarios are summarized in Table 1.
We mainly focus on the standard scenario
CI in this paper, while the experimental results on DI, MI, and SMI are reported to demonstratethe flexibility of the proposed method.
Each scenario contains T = 3 time-steps of the continuouslearning stage.
Details of the scenarios could be found in supplementary materials.
Implementation Details .
For all experiments, we train the model 100 epochs using L BCE + L SDloss and 100 epochs using L BCE + L SD + L PLL loss for each time-step t .
The α for EMA isset to 0.99.
The novelty detection threshold ϵ is set to 0.6.
15 neighbors are selected for eachsample during sample sifting and the top 50% samples with larger local density are sifted out.
Thetemperature τ used in L PLL is set to 0.1.
Mean Squared Error (MSE) Consistency Loss L MSE =
1 N t (cid:80) N t i || ϕ
t D ( x t i ) − ϕ t D ( x ′ i t ) ||
2 2 is also used during the continuous learning following AutoNovel [6],where x ′ it denotes the augmentation of x ti .
More details could be found in supplementary materials.
4.1 Comparisons with SOTA methods on CI scenario
We first investigate the performance of the proposed method under CI scenario.
CIFAR-100, CUB200, ImageNet-100, Stanford-Cars, FGVC-Aircraft, and ImageNet-200 are used in this experiment.
The maximum forgetting M f and final discovery M d are reported in Table 2.
Since there are fewworks about continuous novel category discovery and recent works about incremental learning maynot fit the unsupervised scenario, we compare the proposed method with the lower bound of M fand the upper bound of M d , two recent methods for novel category discovery, i.e. , AutoNovel [6]and DRNCD [15], and the combinations of the novel category discovery methods and continuouslearning methods.
It is noticed that there is not an exact lower/upper bound method under the CI scenario.
Therefore,we conduct two methods under the offline setting , where all of the data containing both known andnovel categories are available in a single stage.
We present the results of K-means [20] as a referenceto the lower bound of M f , and the original AutoNovel method as the upper bound of M d .
It should
[Table]
[Figure]
be enhanced that the lower/upper bound results are for reference and could not exactly representsthe best performance on both M f and M d .
Since recent novel category discovery and incrementallearning methods usually contain multiple stages, we chose AutoNovel and LwF [13] and combinethem for comparison based on the flexibility of implementation.
It is observed that AutoNovel with
LwF greatly improves the M f and M f compared to the online version of AutoNovel.
However,such a simple combination not completely solves the feature conflict caused by the two differenttasks.
The proposed GM framework divides and conquers the conflict of features and achieves thebest results among the compared methods (that conduct classification and discovery training in onesingle phase) on all datasets, indicating the effectiveness of the two phases of growing and merging.
[Table] [Table]
4.2 Extended Experiments on More Complicated Scenarios
In this part, we extend GM to more complicated scenarios: DI, MI, and SMI scenarios.
Experimentsare conducted on CIFAR-100 and the results are listed in Table 3 and Table 4.
Due to there is no novelcategory in the incremental unlabeled data under DI, we remove the cluster head in the AutoNovelto form compared method AutoNovel ∗ .
Incremental learning method iCaRL [14] combined withpseudo label assignment and unsupervised learning method DeepClustering combined with finetuningon incremental data are also chosen for comparisons.
As shown in Table 3, after finetuning on theincremental unlabeled data, all comparison methods show the catastrophic forgetting effect, andthe performances are even worse than the original model.
GM avoids catastrophic forgetting andeffectively utilizes continuous unlabeled data, further improving the final performance.
In MI and
SMI scenarios, incremental data are mixed with known and novel categories, which makes labelassignment for incremental data more difficult.
Nonetheless, as shown in Table 4, GM can stilleffectively avoid catastrophic forgetting and achieve superior performance (especially on M f ) withthe help of two phases of growing and merging, compared to the comparison methods.
4.3 Ablation Studies
Previous experimental results indicate Grow and Merge framework can better accomplish the continuous learning and the novel category discovery in multiple scenarios.
Here we provide intra- andinter- class distance among features and the visualizations in different time of GM in Figure 4.
Itcan be clearly observed that with the alternation of the growing and merging phases, the intra-classdistance first increases and then decreases.
The classes are relatively compact but overlapped withothers before growing phase, loose after growing phase, and compact and separated with others aftermerging phase.
The results are consistent to our motivation that the GM relaxes and tightens thefeatures alternatively to resolve the feature conflicts caused by CCD task.
The detailed analysis forall proposed key components of GM are provided.
Ablation studies are conducted on the CIFAR-100dataset under CI scenario, shown in Table 5 and Tabel 6.
We also provide the experiments aboutestimating the number of novel categories in the supplementary material.
Growing Phase .
We investigate the performance of introducing additional methods for representationlearning in the growing phase.
Combining with cross entropy loss (CE) of the exemplars degrades
M d , which validates that relaxing the constraint of memorizing previous category information canfacilitate category discovery.
Thus, we further introduce more relaxed self-supervised methods.
Combing SimCLR [3] also degrades M d , which could be attributed that pushing the representationof negative samples away hurts the performance.
An alternative method SimSiam [24] achievescomparable performance without pushing negative samples, which also validates the importance of arelaxed representation in this phase.
Merging Phase .
In the merge phase, PLL is employed to improve the representation.
We evaluate theperformance of the models replacing PLL with other representation learning methods.
CE representscross entropy loss with the pseudo label given by the prototypes and SSL denotes standard InfoNCEloss [3, 4].
The experimental results demonstrate that, though these alternative methods could performcompetitively on known classes, they sacrifice the performance of discovering novel classes.
Model Merging .
We evaluate the performance of the model without EMA.
M f increases for55% and M d decreases for 14%, which demonstrates the necessity of EMA.
We also consider analternative fuse method which assigns ϕ S ← ϕ tD after the update of ϕ tD by Eq. 4.
M d is significantlyaffected with a decrease of 33%.
Therefore, we use the fixed ϕ S in this paper.
Static-Dynamic Branch .
The multi-branch aims to maintain known category information anddiscover novel categories on different branches.
We first evaluate the performance when directlyreplacing the multi-branch with a single one.
The results show that M f increases dramatically, whichdemonstrates the importance of the multi-branch to preserve already learned categories.
We furtherinvestigate the components of the static-dynamic branch.
Removing the initialization of branch ϕ
Dalso degrades both M f and M d .
For the proposed loss L SD aligning the feature space of ϕ S and ϕ tD ,results show that the model without L SD performs worse on both M f and M d for the separation ofthe feature spaces leading to the failure of the prototype mechanism.
The model replacing L SD with
DINO [25] increases M f for 3% and decreases M d for 6.3%.
Sifting in Merging .
Sifting aims to sift out samples with low cluster confidence given by the clusterhead.
Except for the criterion based on the adopted local geometry property, there are several potentialcriteria to sift samples.
Confidence Ranking (CR) selects a portion of all samples with the highestscore .
Conditional Confidence Ranking (CCR) selects a portion of samples from each class withthe highest score.
Confidence Filtering (CF) selects the samples with the score higher than a giventhreshold.
The results show that: (i) For M f , the model without sifting and CR performs worse, and
CCR, CF and the local sample density based method performs at almost same level.
(ii)
For M d , thelocal sample density based method outperforms all of these compared methods.
Hyperparameter Analysis of Static-Dynamic Branch .
We also evaluate the effect of α in Eq.
4,shown in Figure 5.
The hyper-parameter α controls the importance of ϕ S relative to ϕ tD .
For themodel with only EMA, both M f and M d decrease with the increase of α , since the model with higherα maintains more information from the known categories.
The model with only L SD ( i.e. , the modelw/o. EMA shown in Tabel 5) performs worse than the proposed full model, which demonstrates thatonly the constrain of the distance between the feature space of ϕ S and ϕ tD is not enough to transferthe knowledge from ϕ S to ϕ tD .
The performances of the models with both EMA and L SD variessmoother, compared with the models with only EMA.
Therefore, the models could achieve higherperformances and be more robust by combining EMA and L SD .
[Table] [Table]
Related Work
[Table] Novel Class Discovery [6–8, 26] aims to discover unknown classes from unlabeled data by leveragingpairwise similarity [6, 27] or employing unsupervised clustering [12, 28].
Most of these methodsassume that there is a pure unlabeled set that only contains samples from unknown categories, whichmeans the class space of unlabeled data is completely disjoint with the labeled set.
And at test time,samples are all from novel classes.
These assumptions limit their applications in the real world.
CCDaims to tackle novel class discovery under a more realistic situation in which the unlabeled data (i)are continuously confronted and (ii) contain both known and unknown classes.
Continual Learning needs the model to extend its ability to address new tasks when confronted withnew data [29–37].
The main challenge of continual learning is to avoid catastrophic forgetting [38, 39],i.e. , the performance degradation on previously learned tasks when new ones are ingested.
Existingmethods can be categorized into replay methods [14, 40, 41], regularization-based methods [13, 42,43 ] and parameter isolation methods [44–46].
Most of the methods assume there are available labeleddata in each time-step.
Nevertheless, the data in continuous streams are more likely to be unlabeledin reality, which means the category information is unknown and needs to be discovered.
Different from existing novel category discovery and continual learning, CCD requires the model todiscover novel categories from the continuous unlabeled data while maintaining the classificationperformance on known and discovered classes.
Conclusion
In this paper, we study the continuous category discovery (CCD) problem where unlabeled data arecontinuously fed into the category discovery system.
The main challenge of the CCD problem is thatdifferent sets of features are needed to classify known categories and discover novel categories.
Toaddress this challenge, we develop a framework called Grow and Merge that alternatively updates themodel in the growing phase and the merging phase.
Extensive experimental results demonstrate that
GM outperforms the existing methods with the less forgetting effect of known categories and higheraccuracy for category discovery.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See supplementary materials.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] See
supplementary materials.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
Theoretical
results are not included.
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main ex-
perimental results (either in the supplemental material or as a URL)?
[Yes] See the supplementary materials.
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes] See section 4 and supplementary materials.
(c) Did you report error bars (e g., with respect to the random seed after running experi-
ments multiple times)?
[No]
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See supplementary materials.
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] See Section 4.
(b) Did you mention the license of the assets?
[N/A]
The license of the assets are not
provided on their official website.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[No]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[No]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A] Crowdsourcing and human subjects are not used in this research.
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]