Original: O
Revised: O

Original: PEN
Revised: PEN

Original: -W ORLD S EMI -S
Revised: -W ORLD S EMI -S

Original: UPERVISED
Revised: UPERVISED

Original: L EARNING
Revised: L EARNING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Supervised and semi-supervised learning methods have been traditionally designed for the closed-world setting which is based on the assumption that unlabeled test data contains only classes previously encountered in the labeled training data.
Revised: Supervised and semi-supervised learning methods have been traditionally designed for the closed-world setting which is based on the assumption that unlabeled test data contains only classes previously encountered in the labeled training data.

Original: However, the real world is often open and dynamic, and thus novel previously unseen classes may appear in the test data or during the model deployment.
Revised: However, the real world is often open and dynamic, and thus novel previously unseen classes may appear in the test data or during the model deployment.

Original: Here, we introduce a new open-world semi-supervised learning setting in which the model is required to recognize previously seen classes, as well as to discover novel classes never seen in the labeled dataset.
Revised: Here, we introduce a new open-world semi-supervised learning setting in which the model is required to recognize previously seen classes, as well as to discover novel classes never seen in the labeled dataset.

Original: To tackle the problem, we propose
Revised: To tackle the problem, we propose

Original: ORCA, an approach that jointly learns a feature representation and a classiﬁer on the labeled and unlabeled subsets of the data.
Revised: ORCA, an approach that jointly learns a feature representation and a classiﬁer on the labeled and unlabeled subsets of the data.

Original: The key idea in ORCA is in introducing uncertainty based adaptive margin that effectively circumvents the bias caused by the imbalance of variance between seen and novel classes.
Revised: The key idea in ORCA is in introducing uncertainty based adaptive margin that effectively circumvents the bias caused by the imbalance of variance between seen and novel classes.

Original: We demonstrate that ORCA accurately discovers novel classes and assigns samples to previously seen classes on standard benchmark image classiﬁcation datasets, including
Revised: We demonstrate that ORCA accurately discovers novel classes and assigns samples to previously seen classes on standard benchmark image classiﬁcation datasets, including

Original: CIFAR and ImageNet.
Revised: CIFAR and ImageNet.

Original: Remarkably, despite solving the harder task ORCA outperforms semi-supervised methods on seen classes, as well as novel class discovery methods on unseen classes, achieving 7% and 151% improvements on seen and unseen classes of the ImageNet dataset.
Revised: Remarkably, despite solving the harder task ORCA outperforms semi-supervised methods on seen classes, as well as novel class discovery methods on unseen classes, achieving 7% and 151% improvements on seen and unseen classes of the ImageNet dataset.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: With the advent of deep learning, remarkable breakthroughs have been achieved and current machine learning systems excel on tasks with large quantities of labeled data.
Revised: With the advent of deep learning, remarkable breakthroughs have been achieved and current machine learning systems excel on tasks with large quantities of labeled data.

Original: Despite the strengths, the vast majority of models are designed for the closed-world setting rooted in the assumption that training and test data come from the same set of predeﬁned classes.
Revised: Despite the strengths, the vast majority of models are designed for the closed-world setting rooted in the assumption that training and test data come from the same set of predeﬁned classes.

Original: This assumption, however, rarely holds in practice, as labeling data depends on the domain-speciﬁc knowledge which can be severely incomplete and insufﬁcient to account for all possible scenarios.
Revised: This assumption, however, rarely holds in practice, as labeling data depends on the domain-speciﬁc knowledge which can be severely incomplete and insufﬁcient to account for all possible scenarios.

Original: Thus, it is unrealistic to expect that one can identify and prelabel all categories/classes ahead of time, and manually supervise machine learning models.
Revised: Thus, it is unrealistic to expect that one can identify and prelabel all categories/classes ahead of time, and manually supervise machine learning models.

Original: In contrast to the commonly assumed closed world, the real world is inherently dynamic and open — new classes can emerge in the test data that have never been encountered during training.
Revised: In contrast to the commonly assumed closed world, the real world is inherently dynamic and open — new classes can emerge in the test data that have never been encountered during training.

Original: Openworld setting requires the models to be able to classify previously seen classes, but also effectively handle never-before-seen classes.
Revised: Openworld setting requires the models to be able to classify previously seen classes, but also effectively handle never-before-seen classes.

Original: This task is very natural to human intelligence; children can effortlessly recognize previously learnt concepts, but also detect the patterns and differences of the new ones.
Revised: This task is very natural to human intelligence; children can effortlessly recognize previously learnt concepts, but also detect the patterns and differences of the new ones.

Original: However, it is still an open question whether we can design versatile models that can successfully deal with the world of unknown, while not forgetting the world of known.
Revised: However, it is still an open question whether we can design versatile models that can successfully deal with the world of unknown, while not forgetting the world of known.

Original: Semi-supervised learning (SSL) (Chapelle et al., 2009) aims in leveraging unlabeled data when labels are difﬁcult and costly to obtain.
Revised: Semi-supervised learning (SSL) (Chapelle et al., 2009) aims in leveraging unlabeled data when labels are difﬁcult and costly to obtain.

Original: Recent works (Oliver et al., 2018; Chen et al., 2020b) show that incorporating novel classes in the unlabeled set degrades performance of SSL methods.
Revised: Recent works (Oliver et al., 2018; Chen et al., 2020b) show that incorporating novel classes in the unlabeled set degrades performance of SSL methods.

Original: To alleviate this limitation, Guo et al.
Revised: To alleviate this limitation, Guo et al.

Original: (2020) ensure safety of SSL with the presence of novel classes as well.
Revised: (2020) ensure safety of SSL with the presence of novel classes as well.

Original: However, the ability to differentiate between seen and unseen classes is not sufﬁcient as we need methods that can properly handle unseen classes.
Revised: However, the ability to differentiate between seen and unseen classes is not sufﬁcient as we need methods that can properly handle unseen classes.

Original: On the other hand, methods for discovering novel classes (Hsu et al., 2018; 2019; Han et al., 2019; 2020) utilize labeled data solely to learn a richer representation, and are not able to recognize seen and discover unseen classes at the same time.
Revised: On the other hand, methods for discovering novel classes (Hsu et al., 2018; 2019; Han et al., 2019; 2020) utilize labeled data solely to learn a richer representation, and are not able to recognize seen and discover unseen classes at the same time.

Original: [Figure]
Revised: [Figure]

Original: Here, we introduce open-world semi-supervised learning.
Revised: Here, we introduce open-world semi-supervised learning.

Original: In this setting, the unlabeled dataset may contain classes that have never been seen in the labeled dataset, and the model needs to be able to: (i) recognize when a sample from the unlabeled data belongs to one of the seen classes present in the labeled dataset, and (ii) automatically discover novel/unseen classes without any previous knowledge (Figure 1).
Revised: In this setting, the unlabeled dataset may contain classes that have never been seen in the labeled dataset, and the model needs to be able to: (i) recognize when a sample from the unlabeled data belongs to one of the seen classes present in the labeled dataset, and (ii) automatically discover novel/unseen classes without any previous knowledge (Figure 1).

Original: The latter requires the ability to identify new features that can separate unseen classes.
Revised: The latter requires the ability to identify new features that can separate unseen classes.

Original: Open-world SSL is related, but differs from, continual learning (Kirkpatrick et al., 2017), generalized zero-shot learning (Xian et al., 2017), open-set (Scheirer et al., 2012) and open-world recognition (Bendale & Boult, 2015).
Revised: Open-world SSL is related, but differs from, continual learning (Kirkpatrick et al., 2017), generalized zero-shot learning (Xian et al., 2017), open-set (Scheirer et al., 2012) and open-world recognition (Bendale & Boult, 2015).

Original: In particular, by utilizing both labeled and unlabeled data open-world SSL relies on the transductive inference unlike the continual learning which sequentially learns new tasks while trying to mitigate catastrophic forgetting.
Revised: In particular, by utilizing both labeled and unlabeled data open-world SSL relies on the transductive inference unlike the continual learning which sequentially learns new tasks while trying to mitigate catastrophic forgetting.

Original: In contrast to the generalized zeroshot learning, open-world SSL does not assume prior knowledge about the unseen classes.
Revised: In contrast to the generalized zeroshot learning, open-world SSL does not assume prior knowledge about the unseen classes.

Original: Finally, opposed to the open-set and open-world recognition, it requires separation of unknown classes and does not need any external supervision, respectively.
Revised: Finally, opposed to the open-set and open-world recognition, it requires separation of unknown classes and does not need any external supervision, respectively.

Original: To address the challenges of open-world SSL, we propose ORCA, an approach that can discover novel classes, while at the same time achieves high performance on classifying previously seen classes.
Revised: To address the challenges of open-world SSL, we propose ORCA (open-world with uncertainty based adaptive margin), an approach that can discover novel classes, while at the same time achieves high performance on classifying previously seen classes.

Original: Using both labeled and unlabeled data, ORCA learns a joint embedding function parameterized by a convolutional neural network, and linear classiﬁer that assigns samples to previously seen classes, or to novel classes discovered by ORCA.
Revised: Using both labeled and unlabeled data, ORCA learns a joint embedding function parameterized by a convolutional neural network, and linear classiﬁer that assigns samples to previously seen classes, or to novel classes discovered by ORCA.

Original: The starting point is to initialize the model using self-supervised pretraining which previously showed effectiveness for semi-supervised learning (Zhai et al., 2019) and novel class discovery (Han et al., 2020; Van Gansbeke et al., 2020).
Revised: The starting point is to initialize the model using self-supervised pretraining which previously showed effectiveness for semi-supervised learning (Zhai et al., 2019) and novel class discovery (Han et al., 2020; Van Gansbeke et al., 2020).

Original: The objective function in ORCA consists of three main components: (i) supervised loss on labeled data, (ii) unsupervised loss on  unlabeled data estimated using pseudo-labels inferred from the most conﬁdent pairwise similarities, and (iii) regularization that prevents the model to assign all unlabeled samples to the same class.
Revised: The objective function in ORCA consists of three main components: (i) supervised loss on labeled data, (ii) pairwise loss on labeled and unlabeled data estimated using pseudo-labels inferred from the most conﬁdent pairwise similarities, and (iii) regularization that prevents the model to assign all unlabeled samples to the same class.

Original: However, naively combining supervised and unsupervised losses leads to the bias towards seen classes which reduces the ability to adapt to novel classes.
Revised: However, naively combining supervised and pairwise losses leads to the bias towards seen classes which reduces the ability to adapt to novel classes.

Original: To mitigate the bias, the key idea in ORCA lies in introducing uncertainty based adaptive margin in the supervised loss that gradually decreases plasticity and increases discriminability of the model during training.
Revised: To mitigate the bias, the key idea in ORCA lies in introducing uncertainty based adaptive margin in the supervised loss that gradually decreases plasticity and increases discriminability of the model during training.

Original: We evaluate ORCA on benchmark image classiﬁcation datasets.
Revised: We evaluate ORCA on benchmark image classiﬁcation datasets.

Original: The results show that ORCA signiﬁcantly outperforms SSL methods on the task of recognizing previously seen classes, as well as novel class discovery methods on the task of discovering unseen classes.
Revised: The results show that ORCA signiﬁcantly outperforms SSL methods on the task of recognizing previously seen classes, as well as novel class discovery methods on the task of discovering unseen classes.

Original: On the latter task, ORCA improves performance of baseline methods by 51% on CIFAR-100 and 151% on ImageNet-100 dataset.
Revised: On the latter task, ORCA improves performance of baseline methods by 51% on CIFAR-100 and 151% on ImageNet-100 dataset.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Open-world SSL lies on the intersection of semi-supervised learning, novel class discovery and open-world recognition.
Revised: Open-world SSL lies on the intersection of semi-supervised learning, novel class discovery and open-world recognition.

Original: Semi-supervised learning (SSL).
Revised: Semi-supervised learning (SSL).

Original: While the literature on SSL (Chapelle et al., 2009) is vast, two most explored directions are to utilize the structure of the unlabeled data using consistency regularization (Sajjadi et al., 2016; Laine & Aila, 2016), or entropy minimization (Grandvalet & Bengio, 2005).
Revised: While the literature on SSL (Chapelle et al., 2009) is vast, two most explored directions are to utilize the structure of the unlabeled data using consistency regularization (Sajjadi et al., 2016; Laine & Aila, 2016), or entropy minimization (Grandvalet & Bengio, 2005).

Original: Closely related to our work are pseudo-labeling based approaches (Lee, 2013;
Revised: Closely related to our work are pseudo-labeling based approaches (Lee, 2013;

Original: Sohn et al., 2020) which generate pseudo-labels for more conﬁdent unlabeled samples and use them as targets in a standard supervised loss function.
Revised: Sohn et al., 2020) which generate pseudo-labels for more conﬁdent unlabeled samples and use them as targets in a standard supervised loss function.

Original: Under the typically assumed closed-world assumption, SSL methods achieve highly competitive performance to supervised methods; however, recent works (Oliver et al., 2018; Chen et al., 2020b) show that including novel classes in the unlabeled set can hurt the performance compared to not using any unlabeled data.
Revised: Under the typically assumed closed-world assumption, SSL methods achieve highly competitive performance to supervised methods; however, recent works (Oliver et al., 2018; Chen et al., 2020b) show that including novel classes in the unlabeled set can hurt the performance compared to not using any unlabeled data.

Original: To mitigate the negative effects,
Revised: To mitigate the negative effects,

Original: DS 3 L (Guo et al., 2020) aims in assigning low weights to samples from novel classes.
Revised: DS 3 L (Guo et al., 2020) aims in assigning low weights to samples from novel classes.

Original: Yet, rejecting samples from novel classes is usually not enough (Boult et al., 2019).
Revised: Yet, rejecting samples from novel classes is usually not enough (Boult et al., 2019).

Original: Thus, open-world SSL aims in solving more realistic and challenging task, requiring from the model to discover novel classes in the unlabeled data and group them into semantically meaningful clusters.
Revised: Thus, open-world SSL aims in solving more realistic and challenging task, requiring from the model to discover novel classes in the unlabeled data and group them into semantically meaningful clusters.

Original: Novel class discovery.
Revised: Novel class discovery.

Original: Novel class discovery, often referred to as cross-task transfer learning (Hsu et al., 2018), is a recently tackled problem related to deep learning based clustering methods (Xie et al., 2016;
Revised: Novel class discovery, often referred to as cross-task transfer learning (Hsu et al., 2018), is a recently tackled problem related to deep learning based clustering methods (Xie et al., 2016;

Original: Yang et al., 2016; 2017; Chang et al., 2017).
Revised: Yang et al., 2016; 2017; Chang et al., 2017).

Original: In contrast to clustering, novel class discovery assumes prior knowledge given in the form of labeled dataset.
Revised: In contrast to clustering, novel class discovery assumes prior knowledge given in the form of labeled dataset.

Original: The task is then to cluster unlabeled dataset consisting of similar, but completely disjoint, classes than those present in the labeled dataset.
Revised: The task is then to cluster unlabeled dataset consisting of similar, but completely disjoint, classes than those present in the labeled dataset.

Original: The main idea is to leverage knowledge of the known classes to improve representation learning on the novel classes.
Revised: The main idea is to leverage knowledge of the known classes to improve representation learning on the novel classes.

Original: Hsu et al.
Revised: Hsu et al.

Original: (2018; 2019) propose to transfer predictive pairwise similarities from labeled to unlabeled data by posing the categorization problem as a surrogate sametask problem.
Revised: (2018; 2019) propose to transfer predictive pairwise similarities from labeled to unlabeled data by posing the categorization problem as a surrogate sametask problem.

Original: Deep Transfer Clustering (Han et al., 2019) extends the deep clustering framework by incorporating information about the known classes.
Revised: Deep Transfer Clustering (Han et al., 2019) extends the deep clustering framework by incorporating information about the known classes.

Original: Han et al.
Revised: Han et al.

Original: (2020) train the model by generating pseudo-labels of the unlabeled data using rank statistics.
Revised: (2020) train the model by generating pseudo-labels of the unlabeled data using rank statistics.

Original: All the aforementioned methods maintain different classiﬁers to generate class/cluster assignments for labeled and unlabeled datasets, which is not applicable in the more difﬁcult open-world SSL that requires the ability to recognize seen and dicover novel classes simultaneously.
Revised: All the aforementioned methods maintain different classiﬁers to generate class/cluster assignments for labeled and unlabeled datasets, which is not applicable in the more difﬁcult open-world SSL that requires the ability to recognize seen and dicover novel classes simultaneously.

Original: Open-set and open-world recognition.
Revised: Open-set and open-world recognition.

Original: Open-set recognition (Scheirer et al., 2012; Geng et al., 2020) considers the scenario in which novel classes can appear during testing, and the model needs to recognize and reject samples of novel classes.
Revised: Open-set recognition (Scheirer et al., 2012; Geng et al., 2020) considers the scenario in which novel classes can appear during testing, and the model needs to recognize and reject samples of novel classes.

Original: Many methods have been proposed to tackle the task such as SVM-based (Scheirer et al., 2012; Jain et al., 2014), distance-based (J ´ unior et al., 2017), and deep learning based methods (Bendale & Boult, 2016).
Revised: Many methods have been proposed to tackle the task such as SVM-based (Scheirer et al., 2012; Jain et al., 2014), distance-based (J ´ unior et al., 2017), and deep learning based methods (Bendale & Boult, 2016).

Original: On the other hand, open-world recognition (Bendale & Boult, 2015; Boult et al., 2019) requires the system to incrementally learn and extend the set of known classes with novel classes.
Revised: On the other hand, open-world recognition (Bendale & Boult, 2015; Boult et al., 2019) requires the system to incrementally learn and extend the set of known classes with novel classes.

Original: Bendale & Boult (2015) gradually label novel classes by human-in-the-loop.
Revised: Bendale & Boult (2015) gradually label novel classes by human-in-the-loop.

Original: Open-world SSL is related to open-world recognition, but leverages unlabeled data in the learning stage and does not need any manual input.
Revised: Open-world SSL is related to open-world recognition, but leverages unlabeled data in the learning stage and does not need any manual input.

Original: Margin loss.
Revised: Margin loss.

Original: Based on the observation that margin term in cross-entropy loss can adjust intra- and inter-class variations, losses like large-margin softmax (Liu et al., 2016), angular softmax (Liu et al., 2017), and additive margin softmax (Wang et al., 2018) have been proposed to increase the interclass margin to achieve better classiﬁcation accuracy.
Revised: Based on the observation that margin term in cross-entropy loss can adjust intra- and inter-class variations, losses like large-margin softmax (Liu et al., 2016), angular softmax (Liu et al., 2017), and additive margin softmax (Wang et al., 2018) have been proposed to increase the interclass margin to achieve better classiﬁcation accuracy.

Original: Cao et al.
Revised: Cao et al.

Original: (2019) assigned different margins to different classes to encourage the optimal trade-off in generalization between frequent and rare classes.
Revised: (2019) assigned different margins to different classes to encourage the optimal trade-off in generalization between frequent and rare classes.

Original: Liu et al.
Revised: Liu et al.

Original: (2020) found that using negative margin can help to enlarge intra-class variance and reduce inter-class variance, leading to the better performance on novel classes in the few-shot learning.
Revised: (2020) found that using negative margin can help to enlarge intra-class variance and reduce inter-class variance, leading to the better performance on novel classes in the few-shot learning.

Original:  M ETHOD 3. P ROBLEM SETTING
Revised:  M ETHOD 3. P ROBLEM SETTING

Original: We ﬁrst formally introduce open-world semi-supervised learning.
Revised: We ﬁrst formally introduce open-world semi-supervised learning.

Original: We assume that a labeled part of the dataset D l = { ( x i , y i ) } mi =1 consisting of m samples with labels, and an unlabeled part of the dataset D u
Revised: We assume that a labeled part of the dataset D l = { ( x i , y i ) } mi =1 consisting of m samples with labels, and an unlabeled part of the dataset D u

Original: = { ( x i ) } ni =1 consisting of n unlabeled samples, are provided during the training phase.
Revised: = { ( x i ) } ni =1 consisting of n unlabeled samples, are provided during the training phase.

Original: We denote the set of ground-truth classes in the labeled data and unlabeled data as C l and C u , respectively.
Revised: We denote the set of ground-truth classes in the labeled data and unlabeled data as C l and C u , respectively.

Original: Novel class discovery assumes that sets of classes in labeled and unlabeled data are disjoint, i.e., C l ∩ C u = ∅ , while (closed-world) SSL by deﬁnition assumes the same set of classes in labeled and unlabeled data, i.e., C l = C u .
Revised: Novel class discovery assumes that sets of classes in labeled and unlabeled data are disjoint, i.e., C l ∩ C u = ∅ , while (closed-world) SSL by deﬁnition assumes the same set of classes in labeled and unlabeled data, i.e., C l = C u .

Original: In contrast, the open-world SSL, we assume C l ∩ C u (cid:54) = ∅ and C l (cid:54) =
Revised: In contrast, the open-world SSL, we assume C l ∩ C u (cid:54) = ∅ and C l (cid:54) =

Original: C u .
Revised: C u .

Original: We consider
Revised: We consider

Original: C s =
Revised: C s =

Original: C l ∩ C u as seen classes, and C n = C u \C l as novel/unseen classes.
Revised: C l ∩ C u as seen classes, and C n = C u \C l as novel/unseen classes.

Original: Given an unlabeled example, open-world SSL requires the algorithm to either (i) correctly classify it as one of the seen classes C s , or (ii) group it with similar samples from unlabeled data to form one of the novel classes C n .
Revised: Given an unlabeled example, open-world SSL requires the algorithm to either (i) correctly classify it as one of the seen classes C s , or (ii) group it with similar samples from unlabeled data to form one of the novel classes C n .

Original: We propose an approach, named ORCA, that effectively addresses the challenges of open-world
Revised: We propose an approach, named ORCA, that effectively addresses the challenges of open-world

Original: SSL.
Revised: SSL.

Original: We ﬁnd that the key challenge is to mitigate the bias towards seen classes caused by learning discriminative representations faster on the seen classes compared to the novel classes, leading to the reduced quality of the estimated pseudo-labels.
Revised: We ﬁnd that the key challenge is to mitigate the bias towards seen classes caused by learning discriminative representations faster on the seen classes compared to the novel classes, leading to the reduced quality of the estimated pseudo-labels.

Original: To circumvent this problem, we introduce uncertainty based adaptive margin in the cross-entropy loss that trades of intra-class and inter-class variance of the seen classes.
Revised: To circumvent this problem, we introduce uncertainty based adaptive margin in the cross-entropy loss that trades of intra-class and inter-class variance of the seen classes.

Original: The training proceeds in two phases.
Revised: The training proceeds in two phases.

Original: First, we pretrain the network using self-supervised pretraining.
Revised: First, we pretrain the network using self-supervised pretraining.

Original: Then, we optimize our proposed objective function consisting of three main components, detailed in the following sections.
Revised: Then, we optimize our proposed objective function consisting of three main components, detailed in the following sections.

Original: 3. S ELF - SUPERVISED PRETRAINING
Revised: 3. S ELF - SUPERVISED PRETRAINING

Original: In the open-world SSL, we need to jointly solve classiﬁcation and clustering tasks.
Revised: In the open-world SSL, we need to jointly solve classiﬁcation and clustering tasks.

Original: Solving clustering task using deep neural networks is a challenging problem that requires learning representation and clustering assignments, resulting in high sensitivity on the network initialization.
Revised: Solving clustering task using deep neural networks is a challenging problem that requires learning representation and clustering assignments, resulting in high sensitivity on the network initialization.

Original: Previous works on deep clustering and novel class discovery pretrained the network using autoencoder (Xie et al., 2016; Guo et al., 2017), or self-supervised learning (Han et al., 2020; Van Gansbeke et al., 2020).
Revised: Previous works on deep clustering and novel class discovery pretrained the network using autoencoder (Xie et al., 2016; Guo et al., 2017), or self-supervised learning (Han et al., 2020; Van Gansbeke et al., 2020).

Original: Pretraining step deﬁnes a prior for the parameter space which provides better initial representations compared to the random initialization.
Revised: Pretraining step deﬁnes a prior for the parameter space which provides better initial representations compared to the random initialization.

Original: Here, we ﬁnd that pretraining step is of essential importance in open-world SSL.
Revised: Here, we ﬁnd that pretraining step is of essential importance in open-world SSL.

Original: Therefore, to obtain more robust representations that can be used to generalize to unseen tasks, we ﬁrst pretrain ORCA using self-supervised learning.
Revised: Therefore, to obtain more robust representations that can be used to generalize to unseen tasks, we ﬁrst pretrain ORCA using self-supervised learning.

Original: Self-supervised learning formulates a pretext/auxiliary task that does not need any manual curation and can be readily applied to both labeled and unlabeled data, such as predicting patch context (Doersch et al., 2015) or image rotation (Gidaris et al., 2018).
Revised: Self-supervised learning formulates a pretext/auxiliary task that does not need any manual curation and can be readily applied to both labeled and unlabeled data, such as predicting patch context (Doersch et al., 2015) or image rotation (Gidaris et al., 2018).

Original: Pretext task guides the model towards learning meaningful representations in a fully unsupervised way.
Revised: Pretext task guides the model towards learning meaningful representations in a fully unsupervised way.

Original: In particular, we rely on the SimCLR (Chen et al., 2020b) approach.
Revised: In particular, we rely on the SimCLR (Chen et al., 2020b) approach.

Original: We pretrain the backbone f θ on the whole dataset D l ∪ D u with a pretext objective.
Revised: We pretrain the backbone f θ on the whole dataset D l ∪ D u with a pretext objective.

Original: Learned representations are then used to initialize the network for a subsequent task.
Revised: Learned representations are then used to initialize the network for a subsequent task.

Original: 3. R ECOGNIZING SEEN AND DISCOVERING NOVEL CLASSES
Revised: 3. R ECOGNIZING SEEN AND DISCOVERING NOVEL CLASSES

Original: In our framework, we propose the objective function that jointly solves supervised classiﬁcation and unsupervised clustering task.
Revised: In our framework, we propose the objective function that jointly solves supervised classiﬁcation and unsupervised clustering task.

Original: Given labeled samples X l = { x i ∈ R N } ni and unlabeled samples
Revised: Given labeled samples X l = { x i ∈ R N } ni and unlabeled samples

Original: X u = { x i ∈ R N } mi , we ﬁrst apply the embedding function f θ : R N → R D , pretrained using selfsupervised learning, to obtain the feature representations Z l = { z i ∈ R D } ni and Z u
Revised: X u = { x i ∈ R N } mi , we ﬁrst apply the embedding function f θ : R N → R D , pretrained using selfsupervised learning, to obtain the feature representations Z l = { z i ∈ R D } ni and Z u

Original: = { z i ∈ R D } mi for labeled and unlabeled data, respectively.
Revised: = { z i ∈ R D } mi for labeled and unlabeled data, respectively.

Original: Here, z i =
Revised: Here, z i =

Original: f θ ( x i ) for every sample x i ∈
Revised: f θ ( x i ) for every sample x i ∈

Original: X l ∪ X u .
Revised: X l ∪ X u .

Original: On top of the pretrained network, we add a classiﬁcation head consisting of a single linear layer parameterized by a weight matrix W : R D
Revised: On top of the pretrained network, we add a classiﬁcation head consisting of a single linear layer parameterized by a weight matrix W : R D

Original: → R |C l | + |C u | , and followed by a softmax layer.
Revised: → R |C l | + |C u | , and followed by a softmax layer.

Original: During training, we freeze the ﬁrst layers of the backbone f θ and update its last layers and classiﬁer W .
Revised: During training, we freeze the ﬁrst layers of the backbone f θ and update its last layers and classiﬁer W .

Original: The ﬁnal class/cluster prediction is calculated as c i = argmax ( W T · z i ) ∈ R .
Revised: The ﬁnal class/cluster prediction is calculated as c i = argmax ( W T · z i ) ∈ R .

Original: Note that if c i / ∈ C l , then x i belongs to novel classes.
Revised: Note that if c i / ∈ C l , then x i belongs to novel classes.

Original: ORCA’s objective consists of three components: (i) unsupervised loss, (ii) supervised loss, and (iii) regularization towards uniform distribution.
Revised: ORCA’s objective consists of three components: (i) pairwise loss, (ii) supervised loss, and (iii) regularization towards uniform distribution.

Original: We assume that the number of novel classes |C u | is known and given as an input to the algorithm.
Revised: We assume that the number of novel classes |C u | is known and given as an input to the algorithm.

Original: This is a typical assumption of clustering and novel class discovery methods.
Revised: This is a typical assumption of clustering and novel class discovery methods.

Original: However, if it is not the case then the number of classes needs to be estimated.
Revised: However, if it is not the case then the number of classes needs to be estimated.

Original: We address this question in the experiments.
Revised: We address this question in the experiments.

Original: 3.3. U NSUPERVISED LOSS
Revised: 3.3. P AIRWISE LOSS

Original: Inspired by (Chang et al., 2017; Hsu et al., 2018), we transform the cluster learning problem into a pairwise similarity prediction task.
Revised: Inspired by (Chang et al., 2017; Hsu et al., 2018), we transform the cluster learning problem into a pairwise similarity prediction task.

Original: Given the labeled dataset X l and unlabeled dataset X u , we aim to ﬁne-tune our embedding function f θ and learn a similarity prediction function parameterized by a linear classiﬁer W such the samples from the same class are grouped together.
Revised: Given the labeled dataset X l and unlabeled dataset X u , we aim to ﬁne-tune our embedding function f θ and learn a similarity prediction function parameterized by a linear classiﬁer W such the samples from the same class are grouped together.

Original: To achieve this, we rely on the ground-truth annotations from the labeled set and pseudo-labels for the unlabeled set.
Revised: To achieve this, we rely on the ground-truth annotations from the labeled set and pseudo-labels for the unlabeled set.

Original: Speciﬁcally, for the labeled set we already know which pairs should belong to the same class so we can use ground-truth labels.
Revised: Speciﬁcally, for the labeled set we already know which pairs should belong to the same class so we can use ground-truth labels.

Original: To obtain the pseudo-labels for the unlabeled set, we calculate the cosine distance between all pairs of feature representations
Revised: To obtain the pseudo-labels for the unlabeled set, we calculate the cosine distance between all pairs of feature representations

Original: z i in a mini-batch.
Revised: z i in a mini-batch.

Original: We then rank the distances and for each sample generate the pseudo-label for its most similar neighbor.
Revised: We then rank the distances and for each sample generate the pseudo-label for its most similar neighbor.

Original: Therefore, we only generate pseudo-labels from the most conﬁdent positive pairs for each sample within the mini-batch.
Revised: Therefore, we only generate pseudo-labels from the most conﬁdent positive pairs for each sample within the mini-batch.

Original: For feature representations Z l ∪Z u in a mini-batch, we denote its closest set as Z (cid:48)
Revised: For feature representations Z l ∪Z u in a mini-batch, we denote its closest set as Z (cid:48)

Original: l ∪Z (cid:48) u .
Revised: l ∪Z (cid:48) u .

Original: Note that Z (cid:48) l is always correct since it is generated using the ground-truth labels.
Revised: Note that Z (cid:48) l is always correct since it is generated using the ground-truth labels.

Original: Unsupervised loss in ORCA is deﬁned as a modiﬁed form of binary cross-entropy loss (BCE):
Revised: Pairwise loss in ORCA is deﬁned as a modiﬁed form of binary cross-entropy loss (BCE):

Original: [Equation]
Revised: [Equation]

Original: where softmax function assigns examples to one of the seen or novel classes.
Revised: where softmax function assigns examples to one of the seen or novel classes.

Revised: For labeled examples, we have the ground truth annotations, so we use them to compute the loss.

Revised: For unlabeled examples, we compute the loss based on the generated pseudo-labels.

Original: The reason behind considering only most conﬁdent pairs  is that we ﬁnd out the increased noise in pseudo-labels is detrimental to cluster learning.
Revised: The reason behind considering only most conﬁdent pairs to generate pseudo-labels is that we ﬁnd that the increased noise in pseudo-labels is detrimental to cluster learning.

Original: Further, unlike (Chang et al., 2017; Hsu et al., 2018; Han et al., 2020)
Revised: Further, unlike (Chang et al., 2017; Hsu et al., 2018; Han et al., 2020)

Original: we consider only positive pairs.
Revised: we consider only positive pairs.

Original: We ﬁnd that including negative pairs in our loss does not beneﬁt learning since the majority of negative pairs can be easily recognized.
Revised: We ﬁnd that including negative pairs in our loss does not beneﬁt learning since the majority of negative pairs can be easily recognized.

Original: Our unsupervised objective with only positive pairs is similar to SCAN (Van Gansbeke et al., 2020).
Revised: Our pairwise objective with only positive pairs is similar to SCAN (Van Gansbeke et al., 2020).

Original: However, we update distances and positive pairs in an online version in order to beneﬁt from the improved feature representation during training.
Revised: However, we update distances and positive pairs in an online version in order to beneﬁt from the improved feature representation during training.

Original: On the other hand, SCAN updates only weights of the linear classiﬁer while freezing feature representation.
Revised: On the other hand, SCAN updates only weights of the linear classiﬁer while freezing feature representation.

Original: 3. S UPERVISED LOSS WITH UNCERTAINTY BASED ADAPTIVE MARGIN
Revised: 3. S UPERVISED LOSS WITH UNCERTAINTY BASED ADAPTIVE MARGIN

Original: For the supervised loss, we utilize the categorical annotations for the labeled data { y i } ni =1 and optimize weights W and backbone θ .
Revised: For the supervised loss, we utilize the categorical annotations for the labeled data { y i } ni =1 and optimize weights W and backbone θ .

Original: First, we propose the baseline for open-world SSL using the standard cross-entropy (CE) loss:
Revised: First, we propose the baseline for open-world SSL using the standard cross-entropy (CE) loss:

Original: [Equation]
Revised: [Equation]

Original: However, using standard cross-entropy on labeled data creates an imbalance problem between the labeled and unlabeled, i.e., gradient is updated for seen classes C s , but not for novel classes C n .
Revised: However, using standard cross-entropy on labeled data creates an imbalance problem between the labeled and unlabeled, i.e., gradient is updated for seen classes C s , but not for novel classes C n .

Original: This can result in learning a classiﬁer with larger magnitudes (Kang et al., 2019) for seen classes, leading the whole model to be biased towards the seen classes.
Revised: This can result in learning a classiﬁer with larger magnitudes (Kang et al., 2019) for seen classes, leading the whole model to be biased towards the seen classes.

Original: To overcome the problem, in ORCA we introduce uncertainty based adaptive margin and propose to normalize logits.
Revised: To overcome the problem, in ORCA we introduce uncertainty based adaptive margin and propose to normalize logits.

Original: Uncertainty based adaptive margin.
Revised: Uncertainty based adaptive margin.

Original: Seen classes are learned faster due to the cross-entropy loss, and consequently they tend to have a smaller intra-class variance compared to the novel classes.
Revised: Seen classes are learned faster due to the cross-entropy loss, and consequently they tend to have a smaller intra-class variance compared to the novel classes.

Original: Since the unsupervised learning loss generates pseudo-labels by ranking distances in the feature space, the imbalance of intra-class variances among classes will result in error-prone pseudo-labels, i.e., samples from novel classes will be assigned to seen classes.
Revised: Since the pairwise loss generates pseudo-labels by ranking distances in the feature space, the imbalance of intra-class variances among classes will result in error-prone pseudo-labels, i.e., samples from novel classes will be assigned to seen classes.

Original: To mitigate this bias, we propose to use adaptive margin to reduce the gap between intra-class variance of the seen and novel classes.
Revised: To mitigate this bias, we propose to use adaptive margin to reduce the gap between intra-class variance of the seen and novel classes.

Original: Intuitively, at the beginning of the training, we want to enforce a larger negative margin to encourage a similarly large intra-class variance of the seen classes with respect to the novel classes.
Revised: Intuitively, at the beginning of the training, we want to enforce a larger negative margin to encourage a similarly large intra-class variance of the seen classes with respect to the novel classes.

Original: Close to the end of training when clusters have been formed for the novel classes, we adjust the margin term to be close to 0 so that useful label information can be fully exploited by the model.
Revised: Close to the end of training when clusters have been formed for the novel classes, we adjust the margin term to be close to 0 so that useful label information can be fully exploited by the model.

Original: Logits normalization.
Revised: Logits normalization.

Original: The unconstrained magnitudes of a classiﬁer can negatively affect the tuning of the margin.
Revised: The unconstrained magnitudes of a classiﬁer can negatively affect the tuning of the margin.

Original: To avoid the problem, we normalize the inputs and weights of the linear classiﬁer, i.e., z i
Revised: To avoid the problem, we normalize the inputs and weights of the linear classiﬁer, i.e., z i

Original: =
Revised: =

Original: z i
Revised: z i

Original: | z i | and W j = W j | W j | .
Revised: | z i | and W j = W j | W j | .

Original: We introduce an additional scaling parameter s that controls the temperature of the cross-entropy loss.
Revised: We introduce an additional scaling parameter s that controls the temperature of the cross-entropy loss.

Original: The design is similar the AM-Softmax (Wang et al., 2018).
Revised: The design is similar the AM-Softmax (Wang et al., 2018).

Original: Finally, supervised loss in ORCA with uncertainty based adaptive margin is deﬁned as follows:
Revised: Finally, supervised loss in ORCA with uncertainty based adaptive margin is deﬁned as follows:

Original: [Equation]
Revised: [Equation]

Original: where ¯ u is uncertainty and λ is a regularizer deﬁning its strength.
Revised: where ¯ u is uncertainty and λ is a regularizer deﬁning its strength.

Original: We set λ to 1 in all our experiments.
Revised: We set λ to 1 in all our experiments.

Original: Uncertainty estimation.
Revised: Uncertainty estimation.

Original: We propose to capture intra-class variance using uncertainty estimated from the conﬁdence of unlabeled samples.
Revised: We propose to capture intra-class variance using uncertainty estimated from the conﬁdence of unlabeled samples computed from the output of the softmax function.

Original: In the binarysetting, ¯ u = 1 |D u | (cid:80) x ∈D u
Revised: In the binary

Original: Var ( Y | X = x ) = 1 |D u | (cid:80) x ∈D u Pr ( Y = 1 | X ) ·
Revised: [Equation]

Original: Pr ( Y = 0 | X ) , which can be further approximated by:
Revised: which can be further approximated by:

Original: [Equation]
Revised: [Equation]

Original: up to a factor of at most 2.
Revised: up to a factor of at most 2.

Original: We use the same formula as an approximation for the group uncertainty in multi-class setting.
Revised: We use the same formula as an approximation for the group uncertainty in multi-class setting.

Original: 3. R EGULARIZATION TOWARDS UNIFORM DISTRIBUTION
Revised: 3. R EGULARIZATION TOWARDS UNIFORM DISTRIBUTION

Original: By using only pairwise loss on the unlabeled data, ORCA could degenerate to a trivial solution of assigning all samples to the same class, i.e., |C u | = 1 .
Revised: By using only pairwise loss on the unlabeled data, ORCA could degenerate to a trivial solution of assigning all samples to the same class, i.e., |C u | = 1 .

Original: To avoid the problem, we introduce KullbackLeibler (KL) divergence term that regularizes Pr ( y | x ∈ D l ∪D l ) to be close to a uniform distribution
Revised: To avoid the problem, we introduce KullbackLeibler (KL) divergence term that regularizes Pr ( y | x ∈ D l ∪D l ) to be close to a uniform distribution

Original: U 
Revised: U :

Original: [Equation]
Revised: [Equation]

Original: This term corresponds to the minimum entropy regularization used in SSL (Grandvalet & Bengio, 2005; Lee, 2013) to prevent the class distribution from being too ﬂat.
Revised: This term corresponds to the minimum entropy regularization used in SSL (Grandvalet & Bengio, 2005; Lee, 2013) to prevent the class distribution from being too ﬂat.

Original: We deﬁne it more generally using KL-divergence, so that if the prior over the classes is known, we can use it instead of the uniform distribution.
Revised: We deﬁne it more generally using KL-divergence, so that if the prior over the classes is known, we can use it instead of the uniform distribution.

Original: 3. F INAL OBJECTIVE
Revised: 3. F INAL OBJECTIVE

Original: The ﬁnal objective function used in ORCA combines unsupervised binary cross-entropy loss, supervised cross-entropy loss with adaptive margin, and proposed regularization:
Revised: The ﬁnal objective function used in ORCA combines pairwise binary cross-entropy loss, supervised cross-entropy loss with adaptive margin, and proposed regularization:

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: where η 1 and η 2 are regularization parameters set to 1 in all our experiments.
Revised: where η 1 and η 2 are regularization parameters set to 1 in all our experiments.

Original: The pseudo-code of the algorithm is summarized in Algorithm 1 in Appendix A.
Revised: The pseudo-code of the algorithm is summarized in Algorithm 1 in Appendix A.

Original:  E XPERIMENTS 4. E XPERIMENTAL SETUP
Revised:  E XPERIMENTS 4. E XPERIMENTAL SETUP

Original: Datasets.
Revised: Datasets.

Original: We evaluate ORCA on standard benchmark image classiﬁcation datasets: CIFAR-10,
Revised: We evaluate ORCA on standard benchmark image classiﬁcation datasets: CIFAR-10,

Original: CIFAR-100 (Krizhevsky, 2009) and ImageNet (Russakovsky et al., 2015) with controllable ratios of unlabeled data and novel classes.
Revised: CIFAR-100 (Krizhevsky, 2009) and ImageNet (Russakovsky et al., 2015) with controllable ratios of unlabeled data and novel classes.

Original: Since the full ImageNet dataset is very large, we sub-sample 100 classes and conduct all experiments on this subset.
Revised: Since the full ImageNet dataset is very large, we sub-sample 100 classes and conduct all experiments on this subset.

Original: For each dataset, we always use the ﬁrst k classes as seen classes, and the rest as novel classes.
Revised: For each dataset, we always use the ﬁrst k classes as seen classes, and the rest as novel classes.

Original: We label 50% samples of the seen classes, and use the rest as unlabeled set.
Revised: We label 50% samples of the seen classes, and use the rest as unlabeled set.

Original: Evaluation metrics.
Revised: Evaluation metrics.

Original: To measure performance on unlabeled data, we follow the evaluation protocol in novel class discovery (Han et al., 2019; 2020).
Revised: To measure performance on unlabeled data, we follow the evaluation protocol in novel class discovery (Han et al., 2019; 2020).

Original: On seen classes, we report accuracy.
Revised: On seen classes, we report accuracy.

Original: On novel classes, we report both accuracy and normalized mutual information (NMI).
Revised: On novel classes, we report both accuracy and normalized mutual information (NMI).

Original: To compute accuracy on the novel classes, we ﬁrst solve optimal assignment problem using Hungarian algorithm (Kuhn, 1955).
Revised: To compute accuracy on the novel classes, we ﬁrst solve optimal assignment problem using Hungarian algorithm (Kuhn, 1955).

Original: When reporting accuracy on all classes jointly, we solve optimal assignment using both seen and novel classes.
Revised: When reporting accuracy on all classes jointly, we solve optimal assignment using both seen and novel classes.

Original: Baselines.
Revised: Baselines.

Original: ORCA is a unique method that can solve both tasks deﬁned in open-world SSL.
Revised: ORCA is a unique method that can solve both tasks deﬁned in open-world SSL.

Original: We compare its performance on seen classes with SSL baselines, and with novel class discovery baselines on novel classes.
Revised: We compare its performance on seen classes with SSL baselines, and with novel class discovery baselines on novel classes.

Original: In particular, we compare performance on seen classes with two representative
Revised: In particular, we compare performance on seen classes with two representative

Original: SSL methods: pseudo-labeling
Revised: SSL methods: pseudo-labeling

Original: (Lee, 2013) and DS 3 L (Guo et al., 2017).
Revised: (Lee, 2013) and DS 3 L (Guo et al., 2020).

Original: DS 3 L adaptively assigns low-weights to samples from unseen classes.
Revised: DS 3 L adaptively assigns low-weights to samples from unseen classes.

Original: On unseen classes, we compare ORCA to two recently proposed novel class discovery methods: DTC (Han et al., 2019) and RankStats (Han et al., 2020).
Revised: On unseen classes, we compare ORCA to two recently proposed novel class discovery methods: DTC (Han et al., 2019) and RankStats (Han et al., 2020).

Original: Since RankStats relies on the self-supervised pretraining but originally pretrains the network using
Revised: Since RankStats relies on the self-supervised pretraining but originally pretrains the network using

Original: RotNet (Gidaris et al., 2018), we pretrained RankStats using SimCLR (Chen et al., 2020a) to ensure that the differences in the performance between ORCA and RankStats are not caused by different pretraining strategy.
Revised: RotNet (Gidaris et al., 2018), we pretrained RankStats using SimCLR (Chen et al., 2020a) to ensure that the differences in the performance between ORCA and RankStats are not caused by different pretraining strategy.

Original: It is important to note that both SSL and novel class discovery methods solve easier tasks compared to ORCA since they only need to recognize seen, or discover novel classes.
Revised: It is important to note that both SSL and novel class discovery methods solve easier tasks compared to ORCA since they only need to recognize seen, or discover novel classes.

Original: We further include comparison to open-world SSL baseline proposed in our work in which adaptive margin cross-entropy in ORCA is replaced with standard cross-entropy loss deﬁned in (2).
Revised: We further include comparison to open-world SSL baseline proposed in our work in which adaptive margin cross-entropy in ORCA is replaced with standard cross-entropy loss deﬁned in (2).

Original: Please refer to Appendix B for implementation details.
Revised: Please refer to Appendix B for implementation details.

Original: We will made code publicly available.
Revised: We will made code publicly available.

Original: 4. R ESULTS
Revised: 4. R ESULTS

Original: Evaluation on benchmark datasets.
Revised: Evaluation on benchmark datasets.

Original: We report results on CIFAR-10, CIFAR-100 and ImageNet100 datasets in Tables 1 and 2.
Revised: We report results on CIFAR-10, CIFAR-100 and ImageNet100 datasets in Tables 1 and 2.

Original: Despite solving harder tasks compared to SSL and novel class discovery methods, ORCA outperforms all baselines on all three datasets, achieving remarkable improvements on novel class discovery task.
Revised: Despite solving harder tasks compared to SSL and novel class discovery methods, ORCA outperforms all baselines on all three datasets, achieving remarkable improvements on novel class discovery task.

Original: In particular, on seen classes ORCA consistently outperforms SSL methods, outperforming DS 3 L by 1 – 7% .
Revised: In particular, on seen classes ORCA consistently outperforms SSL methods, outperforming DS 3 L by 1 – 7% .

Original: Novel class discovery methods can not recognize seen classes, that is match classes in unlabeled dataset to previously seen classes from the labeled dataset.
Revised: Novel class discovery methods can not recognize seen classes, that is match classes in unlabeled dataset to previously seen classes from the labeled dataset.

Original: However, it is possible to evaluate their performance on seen classes by regarding seen classes in the same way as novel classes (denoted by star in the tables).
Revised: However, it is possible to evaluate their performance on seen classes by regarding seen classes in the same way as novel classes (denoted by star in the tables).

Original: By comparing the performance of novel class discovery methods to SSL methods and ORCA, we ﬁnd that they lag far behind.
Revised: By comparing the performance of novel class discovery methods to SSL methods and ORCA, we ﬁnd that they lag far behind.

Original: On novel classes, ORCA achieves improvements over best novel class discovery baseline of 12% and 51% on CIFAR-10 and CIFAR-100 respectively, and 151% and 142% on two splits of
Revised: On novel classes, ORCA achieves improvements over best novel class discovery baseline of 12% and 51% on CIFAR-10 and CIFAR-100 respectively, and 151% and 142% on two splits of

Original: ImageNet-100 dataset.
Revised: ImageNet-100 dataset.

Original: These remarkable improvements show that ORCA is not only unique method in its ability to solve both tasks, but also outperforms other baselines on their respective tasks.
Revised: These remarkable improvements show that ORCA is not only unique method in its ability to solve both tasks, but also outperforms other baselines on their respective tasks.

Original: Furthermore, comparison of ORCA to the proposed open-world SSL baseline clearly demonstrates the importance of introducing uncertainity based adaptive margin in the cross-entropy loss.
Revised: Furthermore, comparison of ORCA to the proposed open-world SSL baseline clearly demonstrates the importance of introducing uncertainty based adaptive margin in the cross-entropy loss.

Revised: We conﬁrm in Appendix C that ORCA’s improvements are retained when the number of labeled examples of seen classes is reduced to only 10% .

Revised: Additional ablation studies on the importance of each component in the objective fumction, sensitivity analysis to parameters

Revised: η 1 and η 2 , and the performance with unbalanced data distributions are reported in Appendix C.

Original:  ORCA and other baselines assume that number of novel classes is known.
Revised: [Table] [Table] Evaluation with the unknown number of novel classes. ORCA and other baselines assume that number of novel classes is known.

Original: If it is not the  case, then the number of classes needs to be estimated ﬁrst.
Revised: However, in the real-world setting we often do not know number of classes in advance. In such case, we can apply ORCA by ﬁrst estimating the number of classes.

Original: We provide results with the estimated number of classes on  CIFAR-100 dataset in Appendix C.
Revised: To evaluate performance on the CIFAR-100 dataset in this scenario, we estimate the number of clusters using technique proposed in DTC (Han et al., 2019) to be 124 .

Original: NMI is reported only for novel classes. We  use 50% classes as seen, and 50%
Revised: We then use the estimated number of classes to re-test all the algorithms.

Original: [Table]
Revised: Results shown in Table 3 show that ORCA outperforms all baselines by a large margin even when the number of novel classes needs to be estimated. Further, with the estimated number of classes ORCA achieves only slightly worse results compared to the setting in which the number of classes is known a priori.

Original: [Table]
Revised: [Table]

Original: Effect of number of novel classes.
Revised: Effect of number of novel classes.

Original: We systematically evaluate performance when varying the ratio of seen and novel classes in the unlabeled set on the CIFAR-100 dataset (Figure 2).
Revised: We systematically evaluate performance when varying the ratio of seen and novel classes in the unlabeled set on the CIFAR-100 dataset (Figure 2).

Original: We ﬁnd that
Revised: We ﬁnd that

Original: ORCA consistently achieves highest accuracy on both seen and novel classes across all values.
Revised: ORCA consistently achieves highest accuracy on both seen and novel classes across all values.

Original: The only exception is DS 3 L approach on seen classes when the number of seen classes is very low.
Revised: The only exception is DS 3 L approach on seen classes when the number of seen classes is very low.

Original: On seen classes, ORCA is the only method that retains stable performance across varying ratio of seen and novel classes.
Revised: On seen classes, ORCA is the only method that retains stable performance across varying ratio of seen and novel classes.

Original: In contrast, pseudo-labelling SSL method signiﬁcantly degrades performance with the large number of seen classes, while DS 3 L degrades performance when seen and novel classes are equally distributed.
Revised: In contrast, pseudo-labelling SSL method signiﬁcantly degrades performance with the large number of seen classes, while DS 3 L degrades performance when seen and novel classes are equally distributed.

Original: On unseen classes, ORCA consistently outperforms baseline and novel class discovery methods by a large margin.
Revised: On unseen classes, ORCA consistently outperforms baseline and novel class discovery methods by a large margin.

Original: [Figure]
Revised: [Figure]

Original: [Figure]

Original: Beneﬁts of uncertainty based adaptive margin.
Revised: Beneﬁts of uncertainty based adaptive margin.

Original: We evaluate the effect of introducing uncertainty based adaptive margin in the cross-entropy loss of ORCA (Figure 3).
Revised: We evaluate the effect of introducing uncertainty based adaptive margin in the cross-entropy loss of ORCA (Figure 3).

Original: We compare ORCA to baseline approach with zero margin, as well as to the ﬁxed negative margin with the value of margin set to 0 .
Revised: We compare ORCA to baseline approach with zero margin, as well as to the ﬁxed negative margin with the value of margin set to 0 .

Original: During training, we report accuracy and uncertainty which captures intra-class variance, deﬁned in equation (4).
Revised: During training, we report accuracy and uncertainty which captures intra-class variance, deﬁned in equation (4).

Original: We ﬁnd that the baseline approach is not able to reduce intra-class variance on novel class during training, resulting in the degraded performance on a novel class discovery task.
Revised: We ﬁnd that the baseline approach is not able to reduce intra-class variance on novel class during training, resulting in the degraded performance on a novel class discovery task.

Original: In contrast, ORCA effectively reduces intra-class variance on both seen and novel classes.
Revised: In contrast, ORCA effectively reduces intra-class variance on both seen and novel classes.

Original: On seen classes, baseline reaches high performance very quickly; however, its accuracy starts to decrease close to the end of training.
Revised: On seen classes, baseline reaches high performance very quickly; however, its accuracy starts to decrease close to the end of training.

Original: On the other hand, ORCA improves accuracy as training proceeds.
Revised: On the other hand, ORCA improves accuracy as training proceeds.

Original: This ﬁnding is in line with the idea that we need to slowly increase discriminability of the model as proposed in ORCA.
Revised: This ﬁnding is in line with the idea that we need to slowly increase discriminability of the model as proposed in ORCA.

Original: While ﬁxed and adaptive margin show similar intra-class variance and accuracy on novel classes, adaptive margin shows clear beneﬁts on seen classes, achieving lower intra-class variance and signiﬁcantly outperforming ﬁxed margin during the whole training process.
Revised: While ﬁxed and adaptive margin show similar intra-class variance and accuracy on novel classes, adaptive margin shows clear beneﬁts on seen classes, achieving lower intra-class variance and signiﬁcantly outperforming ﬁxed margin during the whole training process.

Revised: At the end of training, adaptive margin used in ORCA improves performance over zero margin used in baseline by 21% on seen and 34% on novel classes (Table 1).

Original: Taken together, these results strongly support the importance of the uncertainty based adaptive negative margin.
Revised: Taken together, these results strongly support the importance of the uncertainty based adaptive negative margin.

Original: We inspect the effect of uncertainty strength parameter λ in Appendix C. 
Revised: We inspect the effect of uncertainty strength parameter λ in Appendix C. [Figure]

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We introduced open-world semi-supervised learning (SSL) setting in which the methods need an ability to recognize classes previously encountered in the labeled dataset, as well as discovering novel, never-before-seen classes.
Revised: We introduced open-world semi-supervised learning (SSL) setting in which the methods need an ability to recognize classes previously encountered in the labeled dataset, as well as discovering novel, never-before-seen classes.

Original: To address this problem, we proposed ORCA, an open-world SSL method that effectively trades off intra-class variance with uncertainty based adaptive margin.
Revised: To address this problem, we proposed ORCA, an open-world SSL method that effectively trades off intra-class variance with uncertainty based adaptive margin.

Original: We showed that ORCA signiﬁcantly outperforms SSL baselines on the task of recognizing seen classes and novel class discovery baselines on clustering unseen classes.
Revised: We showed that ORCA signiﬁcantly outperforms SSL baselines on the task of recognizing seen classes and novel class discovery baselines on clustering unseen classes.

Original: ORCA is a unique method that can jointly solve both tasks of open-world SSL.
Revised: ORCA is a unique method that can jointly solve both tasks of open-world SSL.

Original: Our work makes an important step towards designing methods for the more realistic open-world setting.
Revised: Our work makes an important step towards designing methods for the more realistic open-world setting.

Original: A A LGORITHM
Revised: A A LGORITHM

Original: We summarize ORCA algorithm in Algorithm 1.
Revised: We summarize ORCA algorithm in Algorithm 1.

Original: Require:
Revised: Require:

Original: Labeled subset D l = { ( x i , y i ) } mi =1 , unlabeled subset D u = { ( x i ) } ni =1 , number of novel classes, a parameterized backbone f θ , linear classiﬁer with weight W .
Revised: Labeled subset D l = { ( x i , y i ) } mi =1 , unlabeled subset D u = { ( x i ) } ni =1 , number of novel classes, a parameterized backbone f θ , linear classiﬁer with weight W .

Original: 1: Pretrain the model parameters θ with pretext loss 2: for epoch = 1 to E do 3: ¯ u ← EstimateUncertainty ( D u ) 4: for t = 1 to T do 5:
Revised: 1: Pretrain the model parameters θ with pretext loss 2: for epoch = 1 to E do 3: ¯ u ← EstimateUncertainty ( D u ) 4: for t = 1 to T do 5:

Original: X l , X u ← SampleMiniBatch ( D l ∪ D u ) 6:
Revised: X l , X u ← SampleMiniBatch ( D l ∪ D u ) 6:

Original: Z l , Z u ← Forward ( X l ∪ X u ; f θ ) 7:
Revised: Z l , Z u ← Forward ( X l ∪ X u ; f θ ) 7:

Original: Z (cid:48) l , Z (cid:48)
Revised: Z (cid:48) l , Z (cid:48)

Original: u ← FindClosest ( Z l ∪ Z u ) 8:
Revised: u ← FindClosest ( Z l ∪ Z u ) 8:

Original: L BCE ← [Table] m + 
Revised: L BCE ← 1 m + n (cid:80) z

Original: [Table] (cid:80)
Revised: i ,z (cid:48)

Original: [Table] [Table] [Table]
Revised: i ∈ ( Z l ∪Z u , Z (cid:48)

Original:  (cid:104) softmax ( [Table] ) , softmax ( [Table]
Revised: l ∪Z (cid:48) u ) − log (cid:104) softmax ( sW T · z i ) , softmax ( sW T · z (cid:48) i ) (cid:105) 9:

Original: L [Table] ← [Table]
Revised: L CE ← 1 m (cid:80) z i ∈Z l − log e s ( WTyi · zi + λ ¯ u ) e s ( WTy i · zi + λ ¯ u ) + (cid:80) j (cid:54) =

Original: m (cid:80) [Table] − log e [Table]
Revised: i e sWTyj · zi 10:

Original: L R ← KL ( 1 m + n (cid:80) 
Revised: L R ← KL ( 1 m + n (cid:80) z

Original: [Table]

Original: ∪Z

Original: [Table] softmax ( sW [Table] (cid:107)U ( y )) 11: 12: f θ ← SGD with loss L BCE + η 1 L CE + η 2 L R
Revised: i ∈Z l ∪Z u softmax ( sW T · z i ) (cid:107)U ( y )) 11: 12: f θ ← SGD with loss L BCE + η 1 L CE + η 2 L R

Original: B I MPLEMENTATION DETAILS
Revised: B I MPLEMENTATION DETAILS

Original: Our core algorithm is developed using PyTorch (Paszke et al., 2019) and we conduct all the experiments with NVIDIA RTX 2080 Ti.
Revised: Our core algorithm is developed using PyTorch (Paszke et al., 2019) and we conduct all the experiments with NVIDIA RTX 2080 Ti.

Original: Implementation details for CIFAR.
Revised: Implementation details for CIFAR.

Original: We follow the simple data augmentation suggested in (He et al., 2016) with only random crop and horizontal ﬂip.
Revised: We follow the simple data augmentation suggested in (He et al., 2016) with only random crop and horizontal ﬂip.

Original: We use a modiﬁed ResNet-18 that is compatible with input size 32 × 32 following (Han et al., 2020) and repeat all experiments for 3 runs.
Revised: We use a modiﬁed ResNet-18 that is compatible with input size 32 × 32 following (Han et al., 2020) and repeat all experiments for 3 runs.

Original: We train the model using standard SGD with momentum of 0.9, weight decay of 5 × 10 − 4 .
Revised: We train the model using standard SGD with momentum of 0.9, weight decay of 5 × 10 − 4 .

Original: The model is trained for 200 epochs with a batch size of 512.
Revised: The model is trained for 200 epochs with a batch size of 512.

Original: We anneal the learning rate by a factor of 10 at epoch 140 and 180.
Revised: We anneal the learning rate by a factor of 10 at epoch 140 and 180.

Original: Similar to Han et al.
Revised: Similar to Han et al.

Original: (2020), we only update the parameters of the last block of
Revised: (2020), we only update the parameters of the last block of

Original: ResNet in the second training stage to avoid over-ﬁtting.
Revised: ResNet in the second training stage to avoid over-ﬁtting.

Original: We set hyperparameters to the following default values: s = 10 , λ = 1 , η 1 = 1 , η 2 = 1 .
Revised: We set hyperparameters to the following default values: s = 10 , λ = 1 , η 1 = 1 , η 2 = 1 .

Original: They remain the same across all experiments unless otherwise speciﬁed.
Revised: They remain the same across all experiments unless otherwise speciﬁed.

Original: Implementation details for ImageNet.
Revised: Implementation details for ImageNet.

Original: We follow the standard data augmentation including random resized crop and horizontal ﬂip (He et al., 2016).
Revised: We follow the standard data augmentation including random resized crop and horizontal ﬂip (He et al., 2016).

Original: We use ResNet-50 as backbone.
Revised: We use ResNet-50 as backbone.

Original: We train the model using standard SGD with momentum of 0.9, weight decay of 1 × 10 − 4 .
Revised: We train the model using standard SGD with momentum of 0.9, weight decay of 1 × 10 − 4 .

Original: The model is trained for 90 epochs with a batch size of 512.
Revised: The model is trained for 90 epochs with a batch size of 512.

Original: We anneal the learning rate by a factor of 10 at epoch 30 and 60.
Revised: We anneal the learning rate by a factor of 10 at epoch 30 and 60.

Original: Similar to Han et al.
Revised: Similar to Han et al.

Original: (2020), we only update the parameters of the last block of ResNet in the second training stage to avoid over-ﬁtting.
Revised: (2020), we only update the parameters of the last block of ResNet in the second training stage to avoid over-ﬁtting.

Original: We set hyperparameters to the following default values: s = 10 , λ = 1 , η 1 = 1 , η 2 = 1 .
Revised: We set hyperparameters to the following default values: s = 10 , λ = 1 , η 1 = 1 , η 2 = 1 .

Original: They remain the same across all experiments unless otherwise speciﬁed.
Revised: They remain the same across all experiments unless otherwise speciﬁed.

Original: C A DDITIONAL RESULTS
Revised: C A DDITIONAL RESULTS

Revised: Results with less labeled data.

Revised: We compare performance of ORCA and baselines with the reduced number of labeled data.

Revised: Instead of constructing labeled set with 50% examples labeled in seen classes, we evaluate the performance on the labeled set with only 10% labeled examples.

Revised: Results are shown in Table 4.

Original: Sensitivity of uncertainty regularizer λ . 
Revised: We ﬁnd that ORCA’s substantial improvements over SSL methods on seen classes and novel class discovery methods on novel classes are retained. Speciﬁcally, ORCA achieves 7 . 3% and 32 . 2% improvements over DS 3 L on seen classes of CIFAR-10 and CIFAR-100 datasets, respectively. On novel classes, ORCA achieves 33 . 8% and 90 . 4% improvements over RankStats on

Revised: CIFAR-10 and CIFAR-100 datasets, respectively.

Revised: [Table]

Revised: Ablation study on the objective function.

Revised: The objective function in the proposed open-world SSL baseline and ORCA consists of supervised loss, pairwise loss, and regularization towards uniform distribution.

Revised: To investigate importance of each part, we conduct an ablation study in which we modify baseline approach by removing: (i) supervised loss, (ii) regularization towards uniform distribution.

Revised: In the ﬁrst case, we rely only on the regularized pairwise loss to solve the problem, while in the second case we have unregularized supervised and pairwise losses.

Revised: We note that the pairwise loss is essential in order to be able to discover novel classes.

Revised: We show results on CIFAR-10 and

Revised: CIFAR-100 datasets in Table 5.

Revised: Results demonstrate that both supervised loss and regularization are essential parts of the designed objective function and signiﬁcantly improve performance of the open-world SSL baseline, and consequently ORCA.

Revised: report mean accuracy and

Revised: NMI over three runs.

Revised: We use 50% , 50% split for seen and novel classes.

Revised: Dataset

Revised: [Table]

Revised: Sensitivity analysis of η 1 and η 2 .

Revised: Parameters η 1 and η 2 deﬁne importance of the supervised loss and regularization towards uniform distribution, respectively.

Revised: To analyze their effect on the performance, we vary these parameters and evaluate the ORCA’s performance on the CIFAR-100 dataset.

Revised: We ﬁnd that higher values of η 1 achieve slightly better performance on seen classes.

Revised: This result agrees well with the intuition: giving more importance to the supervised loss improves performance on the seen classes.

Revised: The effect of parameter η 2 on seen classes is opposite and lower values of η 2 achieve better performance on seen classes.

Revised: On novel classes, the optimal performance is obtained when η 1 and ηare set to 1 .

Revised: [Table]

Revised: [Table]

Revised: Sensitivity analysis of uncertainty regularizer λ .

Original: The intention of introducing the uncertainty based adaptive margin is to enforce the group of labeled and unlabeled data to have similar intra-class variances.
Revised: The intention of introducing the uncertainty based adaptive margin is to enforce the group of labeled and unlabeled data to have similar intraclass variances.

Original: Here we inspect how does the uncertainty regularizer λ affect performance.
Revised: Here we inspect how does the uncertainty regularizer λ affect performance.

Original: The results are shown in Table 3.
Revised: The results are shown in Table 7.

Original: A slightly larger λ achieves higher accuracy on the novel classes with the cost of lower accuracy on seen classes.
Revised: A slightly larger λ achieves higher accuracy on the novel classes with the cost of lower accuracy on seen classes.

Original: In contrast, smaller values of λ achieve slightly better performance on seen classes.
Revised: In contrast, smaller values of λ achieve slightly better performance on seen classes.

Original: In general, ORCA is robust to the selection of the λ parameter.
Revised: In general, ORCA is robust to the selection of the λ parameter.

Original: [Table]
Revised: [Table]

Original: Results with unknown number of novel classes.
Revised: Ablation study on unbalanced data distribution.

Original: While we follow the previous test protocols and assume the number of novel classes is known for experiments in the main paper, this assumption may be impractical for real-world applications.
Revised: To check whether proposed regularization towards uniform distribution negatively affects performance when the distribution of the classes is unbalanced, we artiﬁcially introduce unbalanced distributions in CIFAR-10 and CIFAR-100 datasets.

Original: As an alternative, we could ﬁrst estimate the number of clusters in  the unlabeled data, which is quite standard approach intraditional clustering when the number of clusters in unknown.
Revised: In particular, we make distributions long-tailed by following an exponential decay in sample sizes across different classes. The imbalance ratio between sample sizes of the most frequent and least frequent class is set to 10 in the experiments.

Original: To estimate the number of clusters, we use the technique proposed in DTC (Han et al., 2019) and use the estimated number of classes to re-test all the algorithms on CIFAR-100 dataset. We estimated the total number of classes to be 124 .
Revised: The results are shown in Table 8. On the unbalanced CIFAR-10 dataset, proposed regularization improves accuracy by 7 . 4% on seen classes and by 35% on novel classes over the non-regularized model. On the unbalanced CIFAR-100 dataset, regularization improves the performance by 16 .

Original: We ﬁnd that ORCA outperforms all baselines by a large margin even when the number of novel classes needs to be estimated (Table 4).
Revised: 9% on seen and 5 .

Original: Further, with estimated number of classes ORCA achieves only slightly worse results compared to the setting in which the number of  classes is known apriori.
Revised: 1% on novel classes. These results demonstrate the advantage of the proposed regularization even when the classes are unbalanced.

Original: [Table]
Revised: [Table]