Original: A UTOMATICALLY D ISCOVERING AND L EARNING N EW V ISUAL C ATEGORIES WITH R ANKING S TATISTICS
Revised: A UTOMATICALLY D ISCOVERING AND L EARNING N EW V ISUAL C ATEGORIES WITH R ANKING S TATISTICS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We tackle the problem of discovering novel classes in an image collection given labelled examples of other classes.
Revised: We tackle the problem of discovering novel classes in an image collection given labelled examples of other classes.

Original: This setting is similar to semi-supervised learning, but signiﬁcantly harder because there are no labelled examples for the new classes.
Revised: This setting is similar to semi-supervised learning, but signiﬁcantly harder because there are no labelled examples for the new classes.

Original: The challenge, then, is to leverage the information contained in the labelled images in order to learn a general-purpose clustering model and use the latter to identify the new classes in the unlabelled data.
Revised: The challenge, then, is to leverage the information contained in the labelled images in order to learn a general-purpose clustering model and use the latter to identify the new classes in the unlabelled data.

Original: In this work we address this problem by combining three ideas: (1) We suggest that the common approach of bootstrapping an image representation using the available labels introduces an unwanted bias, which we avoid by using self-supervised learning to train the representation from scratch on the union of labelled and unlabelled data; (2) We use rank statistics to transfer the model’s knowledge of the labelled classes to the problem of clustering the unlabelled images; Finally, (3) we train the data representation by optimizing a joint objective function on the labelled and unlabelled subsets of the data, improving both the supervised classiﬁcation of the labelled data, and the clustering of the unlabelled, simultaneously.
Revised: In this work we address this problem by combining three ideas: (1) We suggest that the common approach of bootstrapping an image representation using the available labels introduces an unwanted bias, which we avoid by using self-supervised learning to train the representation from scratch on the union of labelled and unlabelled data; (2) We use rank statistics to transfer the model’s knowledge of the labelled classes to the problem of clustering the unlabelled images; Finally, (3) we train the data representation by optimizing a joint objective function on the labelled and unlabelled subsets of the data, improving both the supervised classiﬁcation of the labelled data, and the clustering of the unlabelled, simultaneously.

Original: We evaluate our approach on standard classiﬁcation benchmarks and outperform current methods for novel category discovery by a signiﬁcant margin
Revised: We evaluate our approach on standard classiﬁcation benchmarks and outperform current methods for novel category discovery by a signiﬁcant margin

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: While modern machine learning systems can reach, or even surpass, human level accuracy in certain image classiﬁcation tasks (Deng et al., 2009), this still comes at the cost of collecting large quantities of data annotations for training.
Revised: While modern machine learning systems can reach, or even surpass, human level accuracy in certain image classiﬁcation tasks (Deng et al., 2009), this still comes at the cost of collecting large quantities of data annotations for training.

Original: Recent advances in semi-supervised learning (SSL) (Oliver et al., 2018) have been successful at mixing labelled and unlabelled data, alleviating this issue.
Revised: Recent advances in semi-supervised learning (SSL) (Oliver et al., 2018) have been successful at mixing labelled and unlabelled data, alleviating this issue.

Original: However, these methods still require some annotations for each of the classes that one wishes to learn.
Revised: However, these methods still require some annotations for each of the classes that one wishes to learn.

Original: We argue this is not always possible in real applications.
Revised: We argue this is not always possible in real applications.

Original: For instance, consider the task of recognizing products in supermarkets.
Revised: For instance, consider the task of recognizing products in supermarkets.

Original: Thousands of new products are introduced every week, and it would be very expensive to annotate them all.
Revised: Thousands of new products are introduced every week, and it would be very expensive to annotate them all.

Original: However, new products do not differ drastically from the existing ones, so it should be possible to discover them automatically as they arise in the data.
Revised: However, new products do not differ drastically from the existing ones, so it should be possible to discover them automatically as they arise in the data.

Original: Unfortunately, machines are still unable to effectively learn new classes without manual annotations.
Revised: Unfortunately, machines are still unable to effectively learn new classes without manual annotations.

Original: In this paper, we tackle the  problem  of discovering new visual classes automatically, assuming only that a certain number of classes are already known by the model.
Revised: In this paper, we tackle the recent problem (Hsu et al., 2018; 2019; Han et al., 2019) of discovering new visual classes automatically, assuming only that a certain number of classes are already known by the model.

Original: This knowledge comes in the form of a labelled dataset of images for a certain set of classes.
Revised: This knowledge comes in the form of a labelled dataset of images for a certain set of classes.

Original: Given that this data is labelled, off-theshelf supervised learning techniques can be used to train a very effective classiﬁer for the known classes, particularly if Convolutional Neural Networks (CNNs) are employed.
Revised: Given that this data is labelled, off-the-shelf supervised learning techniques can be used to train a very effective classiﬁer for the known classes, particularly if Convolutional Neural Networks (CNNs) are employed.

Original: However, this does not mean that the learned features are useful as a representation of the new classes .
Revised: However, this does not mean that the learned features are useful as a representation of the new classes .

Original: Furthermore, even if the representation transfers well, one still has the problem of identifying the new classes in an unlabelled dataset, which is a clustering problem.
Revised: Furthermore, even if the representation transfers well, one still has the problem of identifying the new classes in an unlabelled dataset, which is a clustering problem.

Original: We tackle these problems by introducing a novel approach that combines three key ideas (sectionand ﬁg. 1).
Revised: We tackle these problems by introducing a novel approach that combines three key ideas (sectionand ﬁg. 1).

Original: The ﬁrst idea is to pre-train the image representation (a CNN) using all available images, both labelled and unlabelled, using a self-supervised learning objective.
Revised: The ﬁrst idea is to pre-train the image representation (a CNN) using all available images, both labelled and unlabelled, using a self-supervised learning objective.

Original: Crucially, this objective does not leverage the known labels, resulting in features that are much less biased towards the labelled subset.
Revised: Crucially, this objective does not leverage the known labels, resulting in features that are much less biased towards the labelled subset.

Original: Only after pre-training, labels are used to learn a classiﬁer speciﬁc to the labelled data as well as to ﬁne-tune the deepest layers of the CNN, for which self-supervision is not as effective.
Revised: Only after pre-training, labels are used to learn a classiﬁer speciﬁc to the labelled data as well as to ﬁne-tune the deepest layers of the CNN, for which self-supervision is not as effective.

Original: [Figure]
Revised: [Figure]

Original: +
Revised: +

Original: The second idea is a new approach to transfer the model’s knowledge of the labelled classes to the problem of clustering the unlabelled images.
Revised: The second idea is a new approach to transfer the model’s knowledge of the labelled classes to the problem of clustering the unlabelled images.

Original: Information is exchanged by sharing the same representation between labelled and unlabelled images, which is motivated by the fact that the new classes are likely to have at least some amount of visual similarity with the known ones.
Revised: Information is exchanged by sharing the same representation between labelled and unlabelled images, which is motivated by the fact that the new classes are likely to have at least some amount of visual similarity with the known ones.

Original: In more details, pairs of unlabelled images are compared via their representation vectors.
Revised: In more details, pairs of unlabelled images are compared via their representation vectors.

Original: The comparison is done using a robust rank statistics, by testing if two images share the same subset of k maximally activated representation components.
Revised: The comparison is done using a robust rank statistics, by testing if two images share the same subset of k maximally activated representation components.

Original: This test is to decide if two unlabelled images belong to the same (new) class or not, generating a set of noisy pairwise pseudo-labels.
Revised: This test is to decide if two unlabelled images belong to the same (new) class or not, generating a set of noisy pairwise pseudo-labels.

Original: The pseudo-labels are then used to learn a similarity function for the unlabelled images.
Revised: The pseudo-labels are then used to learn a similarity function for the unlabelled images.

Original: The third and ﬁnal idea is, after bootstrapping the representation, to optimise the model by minimizing a joint objective function containing terms for both the labelled and unlabelled subsets, using respectively the given labels and the generated pseudo-labels, thus avoiding the forgetting issue that may arise if a sequential approach was adopted instead.
Revised: The third and ﬁnal idea is, after bootstrapping the representation, to optimise the model by minimizing a joint objective function containing terms for both the labelled and unlabelled subsets, using respectively the given labels and the generated pseudo-labels, thus avoiding the forgetting issue that may arise if a sequential approach was adopted instead.

Original: A further boost is obtained by incorporating incremental learning of the discovered classes in the classiﬁcation task, which allows information to ﬂow between labelled and unlabelled subsets.
Revised: A further boost is obtained by incorporating incremental learning of the discovered classes in the classiﬁcation task, which allows information to ﬂow between labelled and unlabelled subsets.

Original: We evaluate our method on several public benchmarks (section 3), outperforming by a large margin all existing techniques (section 4) that can be applied to this problem, demonstrating the effectiveness of our approach.
Revised: We evaluate our method on several public benchmarks (section 3), outperforming by a large margin all existing techniques (section 4) that can be applied to this problem, demonstrating the effectiveness of our approach.

Original: We conclude the manuscript by summarizing our ﬁndings (section 5).
Revised: We conclude the manuscript by summarizing our ﬁndings (section 5).

Original:  M ETHOD
Revised:  M ETHOD

Original: Given an unlabelled dataset D u = { x ui , i = 1 , . . .
Revised: Given an unlabelled dataset D u = { x ui , i = 1 , . . .

Original: , M } of images x ui ∈ R 3 × H × W , our goal is to automatically cluster the images into a number of classes C u .
Revised: , M } of images x ui ∈ R 3 × H × W , our goal is to automatically cluster the images into a number of classes C u .

Original: The number C u is supposed to be known a-priori.
Revised: The number C u is supposed to be known a-priori.

Original: We also assume to have a second labelled dataset D l = { ( x li , y li ) , i = 1 , . . .
Revised: We also assume to have a second labelled dataset D l = { ( x li , y li ) , i = 1 , . . .

Original: , N } where y li ∈ { 1 , . .
Revised: , N } where y li ∈ { 1 , . .

Original: , C l } is the class label for image x li .
Revised: , C l } is the class label for image x li .

Original: While the statistics of D l and D u differ, we still expect to be able to extract from D l a general notion of what constitutes a “good class” and then use this knowledge to better cluster D u .
Revised: While the statistics of D l and D u differ, we still expect to be able to extract from D l a general notion of what constitutes a “good class” and then use this knowledge to better cluster D u .

Original: We approach the problem by learning an image representation Φ : x (cid:55)→ Φ( x ) ∈ R d in the form of a convolutional network.
Revised: We approach the problem by learning an image representation Φ : x (cid:55)→ Φ( x ) ∈ R d in the form of a convolutional network.

Original: The representation will help us to recognize the known classes and to discover the new ones.
Revised: The representation will help us to recognize the known classes and to discover the new ones.

Original: In order to do so, we combine three ideas, detailed in the next three sections.
Revised: In order to do so, we combine three ideas, detailed in the next three sections.

Original: 2. S ELF - SUPERVISED LEARNING
Revised: 2. S ELF - SUPERVISED LEARNING

Original: Given that we have a certain number of labelled images D l at our disposal, the obvious idea is to use these labels to bootstrap the representation Φ by minimizing a standard supervised objective such as the cross-entropy loss.
Revised: Given that we have a certain number of labelled images D l at our disposal, the obvious idea is to use these labels to bootstrap the representation Φ by minimizing a standard supervised objective such as the cross-entropy loss.

Original: However, experiments show that this causes the representation to overly-specialize for the classes in D l , providing a poor representation of the new classes in D u .
Revised: However, experiments show that this causes the representation to overly-specialize for the classes in D l , providing a poor representation of the new classes in D u .

Original: Thus we resist the temptation of using the labels right away and use instead a self-supervised learning method to bootstrap the representation Φ .
Revised: Thus we resist the temptation of using the labels right away and use instead a self-supervised learning method to bootstrap the representation Φ .

Original: Self-supervised learning has been shown (Kolesnikov et al., 2019; Gidaris et al., 2018) to produce robust low-level features, especially for the ﬁrst few layers of typical CNNs.
Revised: Self-supervised learning has been shown (Kolesnikov et al., 2019; Gidaris et al., 2018) to produce robust low-level features, especially for the ﬁrst few layers of typical CNNs.

Original: It has the beneﬁt that no data annotations are needed, and thus it can be applied to both labelled and unlabelled images during training.
Revised: It has the beneﬁt that no data annotations are needed, and thus it can be applied to both labelled and unlabelled images during training.

Original: This way, we achieve the key beneﬁt of ensuring that the representation is initialized without being biased towards the labelled data.
Revised: This way, we achieve the key beneﬁt of ensuring that the representation is initialized without being biased towards the labelled data.

Original: In detail, we ﬁrst pre-train our model Φ with self-supervision on the union of D l and D u (for which we discard the labels).
Revised: In detail, we ﬁrst pre-train our model Φ with self-supervision on the union of D l and D u (for which we discard the labels).

Original: We use the RotNet (Gidaris et al., 2018) approach 1 due to its simplicity and efﬁcacy, but any self-supervised method could be used instead.
Revised: We use the RotNet (Gidaris et al., 2018) approach 1 due to its simplicity and efﬁcacy, but any self-supervised method could be used instead.

Original: We then extend the pre-trained network Φ with a classiﬁcation head η l : R d → R C l .
Revised: We then extend the pre-trained network Φ with a classiﬁcation head η l : R d → R C l .

Original: In practice, η l is realized as a linear layer followed by a softmax layer.
Revised: In practice, η l is realized as a linear layer followed by a softmax layer.

Original: The composition η l ◦ Φ is ﬁne-tuned on the labelled subset D l of the data in order to obtain a classiﬁcation function for the C l known classes.
Revised: The composition η l ◦ Φ is ﬁne-tuned on the labelled subset D l of the data in order to obtain a classiﬁcation function for the C l known classes.

Original: This is achieved by using the labels
Revised: This is achieved by using the labels

Original: y i and optimizing the standard cross-entropy (CE) loss:
Revised: y i and optimizing the standard cross-entropy (CE) loss:

Original: [Equation]
Revised: [Equation]

Original: where z li =
Revised: where z li =

Original: Φ( x li ) ∈
Revised: Φ( x li ) ∈

Original: R d is the representation of image x li .
Revised: R d is the representation of image x li .

Original: However, in this process only η l and the last macro-block of Φ (section 3) are updated in order to avoid overﬁtting the labelled data.
Revised: However, in this process only η l and the last macro-block of Φ (section 3) are updated in order to avoid overﬁtting the labelled data.

Original: 2. T RANSFER LEARNING VIA RANK STATISTICS
Revised: 2. T RANSFER LEARNING VIA RANK STATISTICS

Original: Once the representation Φ and the classiﬁer η l have been trained, we are ready to look for the new classes in D u .
Revised: Once the representation Φ and the classiﬁer η l have been trained, we are ready to look for the new classes in D u .

Original: Since the classes in D u are unknown, we represent them by deﬁning a relation among pairs of unlabelled images ( x u i , x u j ) .
Revised: Since the classes in D u are unknown, we represent them by deﬁning a relation among pairs of unlabelled images ( x u i , x u j ) .

Original: The idea is that similar images should belong to the same (new) class, which we denote by the symbol s ij = 1 , while dissimilar ones should not, which we denote by s ij = 0 .
Revised: The idea is that similar images should belong to the same (new) class, which we denote by the symbol s ij = 1 , while dissimilar ones should not, which we denote by s ij = 0 .

Original: The problem is then to obtain the labels s ij .
Revised: The problem is then to obtain the labels s ij .

Original: Our assumption is that the new classes will have some degree of visual similarity with the known ones.
Revised: Our assumption is that the new classes will have some degree of visual similarity with the known ones.

Original: Hence, the representation learned from the known classes should, to some extent, be applicable to the new classes as well.
Revised: Hence, the representation learned from the known classes should, to some extent, be applicable to the new classes as well.

Original: As a consequence, we expect the descriptors z ui =
Revised: As a consequence, we expect the descriptors z ui =

Original: Φ( x ui ) and z uj = Φ( x uj ) of two images x ui , x uj from the new classes to be close if they are from the same (new) class, and to be distinct otherwise.
Revised: Φ( x ui ) and z uj = Φ( x uj ) of two images x ui , x uj from the new classes to be close if they are from the same (new) class, and to be distinct otherwise.

Original: Rather than comparing vectors z ui , z uj directly (e.g., by a scalar product), however, we use a more robust rank statistics.
Revised: Rather than comparing vectors z ui , z uj directly (e.g., by a scalar product), however, we use a more robust rank statistics.

Original: Speciﬁcally, we rank the values in vector z ui by magnitude.
Revised: Speciﬁcally, we rank the values in vector z ui by magnitude.

Original: Then, if the rankings obtained for two unlabelled images x ui and x uj are the same, they are very likely to belong to the same (new) class, so we set s ij = 1 .
Revised: Then, if the rankings obtained for two unlabelled images x ui and x uj are the same, they are very likely to belong to the same (new) class, so we set s ij = 1 .

Original: Otherwise, we set s ij = 0 .
Revised: Otherwise, we set s ij = 0 .

Original: In practice, it is too strict to require the two rankings to be identical if the dimension of z ui is high (otherwise we may end up with s ij = 0 for all pairs ( i, j ) , i (cid:54) = j ).
Revised: In practice, it is too strict to require the two rankings to be identical if the dimension of z ui is high (otherwise we may end up with s ij = 0 for all pairs ( i, j ) , i (cid:54) = j ).

Original: Therefore, we relax this requirement by only testing if the sets of the top- k ranked dimensions are the same (e.g., k
Revised: Therefore, we relax this requirement by only testing if the sets of the top- k ranked dimensions are the same (e.g., k

Original: = 5 in our experiments), i.e.:
Revised: = 5 in our experiments), i.e.:

Original: [Equation]
Revised: [Equation]

Original: where top k : R d → P ( { 1 , . . .
Revised: where top k : R d → P ( { 1 , . . .

Original: , d } ) associates to a vector z
Revised: , d } ) associates to a vector z

Original: the subset of indices { 1 , . . .
Revised: the subset of indices { 1 , . . .

Original: , d } of its top- k elements.
Revised: , d } of its top- k elements.

Original: Once the labels s ij have been obtained, we use them as pseudo-labels to train a comparison function for the unlabelled data.
Revised: Once the labels s ij have been obtained, we use them as pseudo-labels to train a comparison function for the unlabelled data.

Original: In order to do this, we apply a new head η u : R d → R C u to the image representation z
Revised: In order to do this, we apply a new head η u : R d → R C u to the image representation z

Original: ui
Revised: ui

Original: =
Revised: =

Original: Φ( x ui ) to extract a new descriptor vector η u ( z ui ) optimized for the unlabelled data.
Revised: Φ( x ui ) to extract a new descriptor vector η u ( z ui ) optimized for the unlabelled data.

Original: As in section 2.1, the head is composed of a linear layer followed by a softmax.
Revised: As in section 2.1, the head is composed of a linear layer followed by a softmax.

Original: Then, the inner product η u ( z ui ) (cid:62) η u ( z uj ) is used as a score for whether images x ui and x uj belong to the same class or not.
Revised: Then, the inner product η u ( z ui ) (cid:62) η u ( z uj ) is used as a score for whether images x ui and x uj belong to the same class or not.

Original: Note that η u ( z ui ) is a normalized vector due to the softmax layer in η u .
Revised: Note that η u ( z ui ) is a normalized vector due to the softmax layer in η u .

Original: This descriptor is trained by optimizing the binary cross-entropy (BCE) loss:
Revised: This descriptor is trained by optimizing the binary cross-entropy (BCE) loss:

Original: [Equation]
Revised: [Equation]

Original: Furthermore, we structure η u in a particular manner: We set its output dimension to be equal to the number of new classes C u .
Revised: Furthermore, we structure η u in a particular manner: We set its output dimension to be equal to the number of new classes C u .

Original: In this manner, we can use the index of the maximum element of each vector ˆ y u i = argmax y
Revised: In this manner, we can use the index of the maximum element of each vector ˆ y u i = argmax y

Original: [ η u ◦ Φ( x u i )]
Revised: [ η u ◦ Φ( x u i )]

Original: y as prediction ˆ y u
Revised: y as prediction ˆ y u

Original: i for the class of image x u i (as opposed to assigning labels via a clustering method such as k -means).
Revised: i for the class of image x u i (as opposed to assigning labels via a clustering method such as k -means).

Original: 2. J OINT TRAINING ON LABELLED AND UNLABELLED DATA
Revised: 2. J OINT TRAINING ON LABELLED AND UNLABELLED DATA

Original: We now have two losses that involve the representation Φ : the CE loss L CE for the labelled data
Revised: We now have two losses that involve the representation Φ : the CE loss L CE for the labelled data

Original: D l and the pairwise BCE loss L BCE for the unlabelled data D u .
Revised: D l and the pairwise BCE loss L BCE for the unlabelled data D u .

Original: They both share the same image embedding Φ .
Revised: They both share the same image embedding Φ .

Original: This embedding can be trained sequentially, ﬁrst on the labelled data, and then on the unlabelled data using the pseudo-labels obtained above.
Revised: This embedding can be trained sequentially, ﬁrst on the labelled data, and then on the unlabelled data using the pseudo-labels obtained above.

Original: However, in this way the model will very likely forget the knowledge learned from the labelled data, which is known as catastrophic forgetting in incremental learning (Rebufﬁ et al., 2017; Lopez-Paz & Ranzato, 2017; Shmelkov et al., 2017;
Revised: However, in this way the model will very likely forget the knowledge learned from the labelled data, which is known as catastrophic forgetting in incremental learning (Rebufﬁ et al., 2017; Lopez-Paz & Ranzato, 2017; Shmelkov et al., 2017;

Original: Aljundi et al., 2018).
Revised: Aljundi et al., 2018).

Original: In contrast, we jointly ﬁnetune our model using both losses at the same time.
Revised: In contrast, we jointly ﬁnetune our model using both losses at the same time.

Original: Note that most of the model Φ is frozen; we only ﬁnetune the last macro-block of Φ together with the two heads η u and η l .
Revised: Note that most of the model Φ is frozen; we only ﬁnetune the last macro-block of Φ together with the two heads η u and η l .

Original: Importantly, as we ﬁnetune the model, the labels s ij are changing at every epoch as the embedding η l is updated.
Revised: Importantly, as we ﬁnetune the model, the labels s ij are changing at every epoch as the embedding η l is updated.

Original: This in turn affects the rank statistics used to determine the labels s ij as explained in section 2.2.
Revised: This in turn affects the rank statistics used to determine the labels s ij as explained in section 2.2.

Original: This leads to a “moving target” phenomenon that can introduce some instability in learning the model.
Revised: This leads to a “moving target” phenomenon that can introduce some instability in learning the model.

Original: This potential issue is addressed in the next section.
Revised: This potential issue is addressed in the next section.

Original: 2. E NFORCING PREDICTIONS TO BE CONSISTENT
Revised: 2. E NFORCING PREDICTIONS TO BE CONSISTENT

Original: In addition to the CE and BCE losses, we also introduce a consistency regularization term, which is used for both labelled and unlabelled data.
Revised: In addition to the CE and BCE losses, we also introduce a consistency regularization term, which is used for both labelled and unlabelled data.

Original: In semi-supervised learning (Oliver et al., 2018;
Revised: In semi-supervised learning (Oliver et al., 2018;

Original: Tarvainen & Valpola, 2017; Laine & Aila, 2017), the idea of consistency is that the class predictions on an image x and on a randomly-transformed counterpart tx (for example an image rotation) should be the same.
Revised: Tarvainen & Valpola, 2017; Laine & Aila, 2017), the idea of consistency is that the class predictions on an image x and on a randomly-transformed counterpart tx (for example an image rotation) should be the same.

Original: In our case, as will be shown in the experiments, consistency is very important to obtain good performance.
Revised: In our case, as will be shown in the experiments, consistency is very important to obtain good performance.

Original: One reason is that, as noted above, the pairwise pseudolabels for the unlabelled data are subject to change on the ﬂy during training.
Revised: One reason is that, as noted above, the pairwise pseudolabels for the unlabelled data are subject to change on the ﬂy during training.

Original: Indeed, for an image x ui and a randomly-transformed counterpart tx ui , if we do not enforce consistency, we can have top k (Φ( x ui ))
Revised: Indeed, for an image x ui and a randomly-transformed counterpart tx ui , if we do not enforce consistency, we can have top k (Φ( x ui ))

Original: (cid:54) =
Revised: (cid:54) =

Original: top k (Φ( tx ui )) .
Revised: top k (Φ( tx ui )) .

Original: According to eq.
Revised: According to eq.

Original: (2) deﬁning s ij , it could result in different s ij for x ui depending on the data augmentation applied to the image.
Revised: (2) deﬁning s ij , it could result in different s ij for x ui depending on the data augmentation applied to the image.

Original: This variability of the ranking labels for a same pair could then perturb the embedding training.
Revised: This variability of the ranking labels for a same pair could then perturb the embedding training.

Original: Following the common practice in semi-supervised learning, we use the Mean Squared Error (MSE) as the consistency cost.
Revised: Following the common practice in semi-supervised learning, we use the Mean Squared Error (MSE) as the consistency cost.

Original: This is given by:
Revised: This is given by:

Original: [Equation]
Revised: [Equation]

Original: where ˆ z is the representation of tx .
Revised: where ˆ z is the representation of tx .

Original: The over all loss of our model can then be written as
Revised: The over all loss of our model can then be written as

Original: [Equation]
Revised: [Equation]

Original: where the coefﬁcient ω ( t ) is a ramp-up function.
Revised: where the coefﬁcient ω ( t ) is a ramp-up function.

Original: This is widely used in semi-supervised learning (Laine & Aila, 2017; Tarvainen & Valpola, 2017).
Revised: This is widely used in semi-supervised learning (Laine & Aila, 2017; Tarvainen & Valpola, 2017).

Original: Following (Laine & Aila, 2017; Tarvainen & Valpola, 2017), we use the sigmoid-shaped function ω ( t )
Revised: Following (Laine & Aila, 2017; Tarvainen & Valpola, 2017), we use the sigmoid-shaped function ω ( t )

Original: =
Revised: =

Original: λe − 5(1 − tT ) 2 , where t is current time step and T is the ramp-up length and λ ∈ R + .
Revised: λe − 5(1 − tT ) 2 , where t is current time step and T is the ramp-up length and λ ∈ R + .

Original: 2. I TERATIVE INCREMENTAL LEARNING
Revised: 2. I TERATIVE INCREMENTAL LEARNING

Original: We also explore a setting analogous to incremental learning.
Revised: We also explore a setting analogous to incremental learning.

Original: In this approach, after tuning on the labelled set (end of section 2.1), we extend the head η l to C u new classes, so that η l : R d →
Revised: In this approach, after tuning on the labelled set (end of section 2.1), we extend the head η l to C u new classes, so that η l : R d →

Original: R C l + C u .
Revised: R C l + C u .

Original: The head parameters for the new classes are initialized randomly.
Revised: The head parameters for the new classes are initialized randomly.

Original: The model is then trained using the same loss eq.
Revised: The model is then trained using the same loss eq.

Original: (5), but the cross-entropy part of the loss is evaluated on both labelled and unlabelled data D l and D u .
Revised: (5), but the cross-entropy part of the loss is evaluated on both labelled and unlabelled data D l and D u .

Original: Since the cross-entropy requires labels, for the unlabelled data we use the pseudo-labels ˆ y u
Revised: Since the cross-entropy requires labels, for the unlabelled data we use the pseudo-labels ˆ y u

Original: i , which are generated on-the-ﬂy from the head η u at each forward pass.
Revised: i , which are generated on-the-ﬂy from the head η u at each forward pass.

Original: The advantage is that this approach increments η l to discriminate both old and new classes, which is often desirable in applications.
Revised: The advantage is that this approach increments η l to discriminate both old and new classes, which is often desirable in applications.

Original: It also creates a feedback loop that causes the features z ui to be reﬁned, which in turn generates better pseudo-labels ˆ y ui for D u from the head η u .
Revised: It also creates a feedback loop that causes the features z ui to be reﬁned, which in turn generates better pseudo-labels ˆ y ui for D u from the head η u .

Original: In this manner, further improvements can be obtained by this cycle of positive interactions between the two heads during training.
Revised: In this manner, further improvements can be obtained by this cycle of positive interactions between the two heads during training.

Original: [Table]
Revised: [Table]

Original: [Table]

Original:  E XPERIMENTS 3. D ATA AND EXPERIMENTAL DETAILS
Revised:  E XPERIMENTS 3. D ATA AND EXPERIMENTAL DETAILS

Original: We evaluate our models on a variety of classic benchmark datasets: CIFAR-10 (Krizhevsky & Hinton, 2009), CIFAR-100 (Krizhevsky & Hinton, 2009), SVHN (Netzer et al., 2011), OmniGlot (Lake et al., 2015), and ImageNet (Deng et al., 2009).
Revised: We evaluate our models on a variety of classic benchmark datasets: CIFAR-10 (Krizhevsky & Hinton, 2009), CIFAR-100 (Krizhevsky & Hinton, 2009), SVHN (Netzer et al., 2011), OmniGlot (Lake et al., 2015) [Table] and ImageNet (Deng et al., 2009).

Original: Following Han et al.
Revised: Following Han et al.

Original: (2019), we split these to have 5/20/5/654/30 classes respectively in the unlabelled set.
Revised: (2019), we split these to have 5/20/5/654/30 classes respectively in the unlabelled set.

Original: In addition for OmniGlot and ImageNet we use 20 and 3 different splits respectively, as in Han et al. 
Revised: In addition for OmniGlot and ImageNet we use 20 and 3 different splits respectively, as in Han et al. [Table]

Original: and report average clustering accuracy.
Revised: and report average clustering accuracy.

Original: More details of the splits can be found in the appendix.
Revised: More details of the splits can be found in the appendix.

Original: Evaluation metrics.
Revised: Evaluation metrics.

Original: We adopt clustering accuracy (ACC) to evaluate the clustering performance of our approach.
Revised: We adopt clustering accuracy (ACC) to evaluate the clustering performance of our approach.

Original: The ACC is deﬁned as follow:
Revised: The ACC is deﬁned as follow:

Original: [Equation]
Revised: [Equation]

Original: where y i and y i denote the ground-truth label and clustering assignment for each data point x ui ∈ D u respectively, and Sym( L ) is the group permutations of L elements (since the cluster assignments may not in the same order with the ground-truth labels).
Revised: where y i and y i denote the ground-truth label and clustering assignment for each data point x ui ∈ D u respectively, and Sym( L ) is the group permutations of L elements (since the cluster assignments may not in the same order with the ground-truth labels).

Original: Permutations are optimized using the
Revised: Permutations are optimized using the

Original: Hungarian algorithm (Kuhn, 1955).
Revised: Hungarian algorithm (Kuhn, 1955).

Original: Implementation details.
Revised: Implementation details.

Original: We use the ResNet-18 (He et al., 2016) architecture, except for OmniGlot for which we use a VGG-like network (Simonyan & Zisserman, 2015) with six layers to make our setting directly comparable to prior work.
Revised: We use the ResNet-18 (He et al., 2016) architecture, except for OmniGlot for which we use a VGG-like network (Simonyan & Zisserman, 2015) with six layers to make our setting directly comparable to prior work.

Original: We use SGD with momentum (Sutskever et al., 2013) as optimizer for all but the OmniGlot dataset, for which we use Adam (Kingma & Ba, 2014).
Revised: We use SGD with momentum (Sutskever et al., 2013) as optimizer for all but the OmniGlot dataset, for which we use Adam (Kingma & Ba, 2014).

Original: For all experiments we use a batch size of 128.
Revised: For all experiments we use a batch size of 128 and k = 5 which we found worked consistently well across datasets (see appendix D).

Original: More details about the hyper-parameters can be found in the appendix.
Revised: More details about the hyper-parameters can be found in the appendix.

Original: 3. A BLATION STUDY
Revised: 3. A BLATION STUDY

Original: We validate the effectiveness of the components of our method by ablating them and measuring the resulting ACC on the unlabelled data.
Revised: We validate the effectiveness of the components of our method by ablating them and measuring the resulting ACC on the unlabelled data.

Original: Note that, since the evaluation is restricted to the unlabelled data, we are solving a clustering problem.
Revised: Note that, since the evaluation is restricted to the unlabelled data, we are solving a clustering problem.

Original: Hence in this experiment there is no distinction between a training and testing phase; nevertheless, the predicted cluster assignments used to compute the reported ACC do not use data augmentation (i.e. image transformations), which is otherwise used to learn the representation.
Revised: Hence in this experiment there is no distinction between a training and testing phase; nevertheless, the predicted cluster assignments used to compute the reported ACC do not use data augmentation (i.e. image transformations), which is otherwise used to learn the representation.

Original: As can be seen in table 1, all components have a signiﬁcant effect as removing any of them causes the performance to drop substantially.
Revised: As can be seen in table 1, all components have a signiﬁcant effect as removing any of them causes the performance to drop substantially.

Original: Among them, the BCE loss is by far the most important one, since removing it results in a dramatic drop of 40–60% absolute ACC points.
Revised: Among them, the BCE loss is by far the most important one, since removing it results in a dramatic drop of 40–60% absolute ACC points.

Original: For example, the full method has ACC 90 .
Revised: For example, the full method has ACC 90 .

Original: 4% on CIFAR-10, while removing BCE causes the ACC to drop to 26 .
Revised: 4% on CIFAR-10, while removing BCE causes the ACC to drop to 26 .

Original: This shows that that our rank-based embedding comparison can indeed generate reliable pairwise pseudo labels for the BCE loss.
Revised: This shows that that our rank-based embedding comparison can indeed generate reliable pairwise pseudo labels for the BCE loss.

Original: Without consistency, cross entropy, or self-supervision, the performance drops by a more modest but still signiﬁcant 7 . 8% , 5 .
Revised: Without consistency, cross entropy, or self-supervision, the performance drops by a more modest but still signiﬁcant 7 . 8% , 5 .

Original: 7% and 1 . 0% absolute ACC points, respectively, for CIFAR-10.
Revised: 7% and 1 . 0% absolute ACC points, respectively, for CIFAR-10.

Original: It means that the consistency term plays a role as important as the cross-entropy term by preventing the “moving target” phenomenon described in section 2.4.
Revised: It means that the consistency term plays a role as important as the cross-entropy term by preventing the “moving target” phenomenon described in section 2.4.

Original: Finally, by incorporating the discovered classes in the classiﬁcation task, we get a further boost of 1.3%, 2.0% and 0.2% points on CIFAR-10, CIFAR-100 and SVHN respectively.
Revised: Finally, by incorporating the discovered classes in the classiﬁcation task, we get a further boost of 1.3%, 2.0% and 0.2% points on CIFAR-10, CIFAR-100 and SVHN respectively.

Original: 3. N OVEL CATEGORY DISCOVERY
Revised: 3. N OVEL CATEGORY DISCOVERY

Original: We compare our method to baselines and state-of-the-art methods for new class discovery, starting from CIFAR-10, CIFAR-100, and SVHN in table 2.
Revised: We compare our method to baselines and state-of-the-art methods for new class discovery, starting from CIFAR-10, CIFAR-100, and SVHN in table 2.

Original: The ﬁrst baseline (row 5 in table 2) amounts
Revised: The ﬁrst baseline (row 5 in table 2) amounts

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: to apply k -means (MacQueen, 1967) to the features extracted by the ﬁne-tuned model (the second step in section 2.1) (we use the k -means++ (Arthur & Vassilvitskii, 2007) initialization).
Revised: to apply k -means (MacQueen, 1967) to the features extracted by the ﬁne-tuned model (the second step in section 2.1) (we use the k -means++ (Arthur & Vassilvitskii, 2007) initialization).

Original: The second baseline (row 1 in table 2) is similar, but uses as feature extractor a model trained from scratch using only the labelled images, which corresponds to a standard transfer learning setting.
Revised: The second baseline (row 1 in table 2) is similar, but uses as feature extractor a model trained from scratch using only the labelled images, which corresponds to a standard transfer learning setting.

Original: By comparing rows 1, 5 and 9 in table 2, we can see that our method substantially outperforms k -means.
Revised: By comparing rows 1, 5 and 9 in table 2, we can see that our method substantially outperforms k -means.

Original: Next, we evaluate the state-of-the-ary methods KCL (Hsu et al., 2018), MCL (Hsu et al., 2019) and DTC (Han et al., 2019).
Revised: Next, we evaluate the state-of-the-ary methods KCL (Hsu et al., 2018), MCL (Hsu et al., 2019) and DTC (Han et al., 2019).

Original: By comparing rows 2–4 to 9, we see that our method outperforms these by a large margin.
Revised: By comparing rows 2–4 to 9, we see that our method outperforms these by a large margin.

Original: We also tried to improve KCL, MCL and DTC by using the same self-supervised initialization we adopt (section 2.1), which indeed results in an improvement (rows 2–4 vs 6–8).
Revised: We also tried to improve KCL, MCL and DTC by using the same self-supervised initialization we adopt (section 2.1), which indeed results in an improvement (rows 2–4 vs 6–8).

Original: However, their overall performance still lags behind ours by a large margin.
Revised: However, their overall performance still lags behind ours by a large margin.

Original: For example, our method achieves 95 . 0% ACC on SVHN, while “KCL w/ S.S.”, “MCL w/ S.S.” and “DTC w/ S.S.” achieve only 65 . 6% , 53 . 1% and 75 .
Revised: For example, our method achieves 95 . 0% ACC on SVHN, while “KCL w/ S.S.”, “MCL w/ S.S.” and “DTC w/ S.S.” achieve only 65 . 6% , 53 . 1% and 75 .

Original: 7% ACC, respectively.
Revised: 7% ACC, respectively.

Original: Similar trends hold for CIFAR-10 and CIFAR-100.
Revised: Similar trends hold for CIFAR-10 and CIFAR-100.

Original: Finally, the iterative incremental learning scheme of section 2.5 results in further improvements, as can be seen by comparing rows 9 and 10 of table 2.
Revised: Finally, the iterative incremental learning scheme of section 2.5 results in further improvements, as can be seen by comparing rows 9 and 10 of table 2.

Original: In ﬁg. 2, we show the evolution of the learned representation on the unlabelled data on CIFARusing t-SNE (van der Maaten & Hinton, 2008).
Revised: In ﬁg. 2, we show the evolution of the learned representation on the unlabelled data on CIFARusing t-SNE (van der Maaten & Hinton, 2008).

Original: As it can be seen, while the clusters overlap in the beginning, they become more and more separated as the training progresses, showing that our model can effectively discover novel visual categories without labels and learn meaningful embeddings for them.
Revised: As it can be seen, while the clusters overlap in the beginning, they become more and more separated as the training progresses, showing that our model can effectively discover novel visual categories without labels and learn meaningful embeddings for them.

Original: We further compare our method to others on two more challenging datasets, OmniGlot and ImageNet, in table 3.
Revised: We further compare our method to others on two more challenging datasets, OmniGlot and ImageNet, in table 3.

Original: For OmniGlot, results are averaged over the 20 alphabets in evaluation set (see supp.
Revised: For OmniGlot, results are averaged over the 20 alphabets in evaluation set (see supp.

Original: matt.); for ImageNet, results are averaged over the three 30-class unlabelled sets used in (Hsu et al., 2018; 2019).
Revised: matt.); for ImageNet, results are averaged over the three 30-class unlabelled sets used in (Hsu et al., 2018; 2019).

Original: Since we have a relatively larger number of labelled classes in these two datasets, we follow (Han et al., 2019) and use metric learning on the labelled classes to pre-train the feature extractor, instead of the self-supervised learning.
Revised: Since we have a relatively larger number of labelled classes in these two datasets, we follow (Han et al., 2019) and use metric learning on the labelled classes to pre-train the feature extractor, instead of the self-supervised learning.

Original: We empirically found that self-supervision does not provide obvious gains for these two datasets.
Revised: We empirically found that self-supervision does not provide obvious gains for these two datasets.

Original: This is reasonable since the data in the labelled sets of these two datasets are rather diverse and abundant, so metric learning can provide good feature initialization since there is less dataset bias.
Revised: This is reasonable since the data in the labelled sets of these two datasets are rather diverse and abundant, so metric learning can provide good feature initialization since there is less dataset bias.

Original: However, by comparing rows 1 and 5 in table 3, it is clear that metric learning alone is not sufﬁcient for the task of novel category discovery.
Revised: However, by comparing rows 1 and 5 in table 3, it is clear that metric learning alone is not sufﬁcient for the task of novel category discovery.

Original: Our method substantially outperforms the k -means results obtained using the features from metric learning — by 11 .
Revised: Our method substantially outperforms the k -means results obtained using the features from metric learning — by 11 .

Original: 9% and 10 .
Revised: 9% and 10 .

Original: 6% on OmniGlot and ImageNet respectively.
Revised: 6% on OmniGlot and ImageNet respectively.

Original: Our method also substantially outperforms the current state-of-the-art, achieving 89 .
Revised: Our method also substantially outperforms the current state-of-the-art, achieving 89 .

Original: 1% and 82 .
Revised: 1% and 82 .

Original: 5% ACC on OmniGlot and ImageNet respectively, compared with 89 . 0% and 78 .
Revised: 5% ACC on OmniGlot and ImageNet respectively, compared with 89 . 0% and 78 .

Original: 8% of (Han et al., 2019), thus setting the new state-of-the-art.
Revised: 8% of (Han et al., 2019), thus setting the new state-of-the-art.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: 3. I NCREMENTAL LEARNING
Revised: 3. I NCREMENTAL LEARNING

Original: Methods for novel category discovery such as (Han et al., 2019; Hsu et al., 2019; 2018) focus on obtaining the highest clustering accuracy for the new unlabelled classes, but may forget the existing labelled classes in the process.
Revised: Methods for novel category discovery such as (Han et al., 2019; Hsu et al., 2019; 2018) focus on obtaining the highest clustering accuracy for the new unlabelled classes, but may forget the existing labelled classes in the process.

Original: In practice, forgetting is not desirable as the model should be able to recognize both old and new classes.
Revised: In practice, forgetting is not desirable as the model should be able to recognize both old and new classes.

Original: Thus, we argue that the classiﬁcation accuracy on the labelled classes should be assessed as well, as for any incremental learning setting.
Revised: Thus, we argue that the classiﬁcation accuracy on the labelled classes should be assessed as well, as for any incremental learning setting.

Original: Note however that our setup differs substantially from standard incremental learning (Rebufﬁ et al., 2017; Lopez-Paz &
Revised: Note however that our setup differs substantially from standard incremental learning (Rebufﬁ et al., 2017; Lopez-Paz &

Original: Ranzato, 2017; Shmelkov et al., 2017; Aljundi et al., 2018) where every class is labelled and the focus is on using limited memory.
Revised: Ranzato, 2017; Shmelkov et al., 2017; Aljundi et al., 2018) where every class is labelled and the focus is on using limited memory.

Original: In our case, we can store and access the original data without memory constraints, but the new classes are unlabelled, which is often encountered in applications.
Revised: In our case, we can store and access the original data without memory constraints, but the new classes are unlabelled, which is often encountered in applications.

Original: By construction (section 2.5), our method learns the new classes on top of the old ones incrementally, out of the box.
Revised: By construction (section 2.5), our method learns the new classes on top of the old ones incrementally, out of the box.

Original: In order to compare to methods such as KCL, MCL and DTC that do not have this property, we proceed as follows.
Revised: In order to compare to methods such as KCL, MCL and DTC that do not have this property, we proceed as follows.

Original: First, the method is run as usual to cluster the unlabelled portion of the data, thus obtaining pseudo-labels for it, and learning as a byproduct a feature extractor.
Revised: First, the method is run as usual to cluster the unlabelled portion of the data, thus obtaining pseudo-labels for it, and learning as a byproduct a feature extractor.

Original: Then, the feature extractor is used to compute features for both the labelled and unlabelled training data, and a linear classiﬁer is trained using labels and pseudo-labels, jointly on all the classes, old and new.
Revised: Then, the feature extractor is used to compute features for both the labelled and unlabelled training data, and a linear classiﬁer is trained using labels and pseudo-labels, jointly on all the classes, old and new.

Original: We report in table 4 the performance of the resulting joint classiﬁer networks on the testing set of each dataset (this is now entirely disjoint from the training set).
Revised: We report in table 5 the performance of the resulting joint classiﬁer networks on the testing set of each dataset (this is now entirely disjoint from the training set).

Original: Our method has similar performances on the old and new classes for CIFAR-10 and SVHN, as might be expected as the split between old and new classes is balanced.
Revised: Our method has similar performances on the old and new classes for CIFAR-10 and SVHN, as might be expected as the split between old and new classes is balanced.

Original: In comparison, the feature extractor learned by KCL and
Revised: In comparison, the feature extractor learned by KCL and

Original: MCL works much better for the old classes (e.g., the accuracy discrepancy between old and new classes is 25 .
Revised: MCL works much better for the old classes (e.g., the accuracy discrepancy between old and new classes is 25 .

Original: 3% for KCL on SVHN).
Revised: 3% for KCL on SVHN).

Original: Conversely, DTC learns features that work better for the new classes, as shown by the poor performance for the old classes on CIFAR-10.
Revised: Conversely, DTC learns features that work better for the new classes, as shown by the poor performance for the old classes on CIFAR-10.

Original: Thus, KCL, MCL and
Revised: Thus, KCL, MCL and

Original: DTC learn representations that are biased to either the old or new classes, resulting overall in suboptimal performance.
Revised: DTC learn representations that are biased to either the old or new classes, resulting overall in suboptimal performance.

Original: In contrast, our method works well on both old and new classes; furthermore, it drastically outperforms existing methods on both.
Revised: In contrast, our method works well on both old and new classes; furthermore, it drastically outperforms existing methods on both.

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: Our work draws inspiration from semi-supervised learning, transfer learning, clustering, and zeroshot learning.
Revised: Our work draws inspiration from semi-supervised learning, transfer learning, clustering, and zeroshot learning.

Original: We review below the most relevant contributions.
Revised: We review below the most relevant contributions.

Original: In semi-supervised learning (SSL) (Chapelle et al., 2006), a partially labelled training dataset is given and the objective is to learn a model that can propagate the labels from the labelled data to unlabelled data.
Revised: In semi-supervised learning (SSL) (Chapelle et al., 2006), a partially labelled training dataset is given and the objective is to learn a model that can propagate the labels from the labelled data to unlabelled data.

Original: Most SSL methods focus on the classiﬁcation task where, usually both labelled and unlabelled points belong to the same set of classes.
Revised: Most SSL methods focus on the classiﬁcation task where, usually both labelled and unlabelled points belong to the same set of classes.

Original: On the contrary, our goal is to handle the case where the unlabelled data classes differ from the labelled data.
Revised: On the contrary, our goal is to handle the case where the unlabelled data classes differ from the labelled data.

Original: Oliver et al.
Revised: Oliver et al.

Original: (2018) summarizes the state-of-the-art SSL methods.
Revised: (2018) summarizes the state-of-the-art SSL methods.

Original: Among them, the consistency-based methods appeared to be the
Revised: Among them, the consistency-based methods appeared to be the

Original: [Figure]
Revised: [Figure]

Original: most effective.
Revised: most effective.

Original: Rasmus et al.
Revised: Rasmus et al.

Original: (2015) propose a ladder network which is trained on both labelled and unlabelled data using a reconstruction loss.
Revised: (2015) propose a ladder network which is trained on both labelled and unlabelled data using a reconstruction loss.

Original: Laine & Aila (2017) simpliﬁes this ladder network by enforcing prediction consistency between a data point and its augmented counterpart.
Revised: Laine & Aila (2017) simpliﬁes this ladder network by enforcing prediction consistency between a data point and its augmented counterpart.

Original: As an alternative to data augmentation, they also consider a regularization method based on the exponential moving average (EMA) of the predictions.
Revised: As an alternative to data augmentation, they also consider a regularization method based on the exponential moving average (EMA) of the predictions.

Original: This idea is further improved by Tarvainen & Valpola (2017): instead of using the EMA of predictions, they propose to maintain the EMA of model parameters.
Revised: This idea is further improved by Tarvainen & Valpola (2017): instead of using the EMA of predictions, they propose to maintain the EMA of model parameters.

Original: The consistency is then measured between the predictions of the current model (student) and the predictions of the EMA model (teacher).
Revised: The consistency is then measured between the predictions of the current model (student) and the predictions of the EMA model (teacher).

Original: Transfer learning (Pan & Yang, 2010; Weiss et al., 2016; Tan et al., 2018) is an effective way to reduce the amount of data annotations required to train a model by pre-training the model on an existing dataset.
Revised: Transfer learning (Pan & Yang, 2010; Weiss et al., 2016; Tan et al., 2018) is an effective way to reduce the amount of data annotations required to train a model by pre-training the model on an existing dataset.

Original: In image classiﬁcation, for example, it is customary to start from a model pretrained on the ImageNet (Deng et al., 2009) dataset.
Revised: In image classiﬁcation, for example, it is customary to start from a model pretrained on the ImageNet (Deng et al., 2009) dataset.

Original: In most transfer learning settings, however, both the source data and the target data are fully annotated.
Revised: In most transfer learning settings, however, both the source data and the target data are fully annotated.

Original: In contrast, our goal is to transfer information from a labelled dataset to an unlabelled one.
Revised: In contrast, our goal is to transfer information from a labelled dataset to an unlabelled one.

Original: Many classic (e.g., Aggarwal & Reddy (2013); MacQueen (1967); Comaniciu & Meer (1979); Ng et al.
Revised: Many classic (e.g., Aggarwal & Reddy (2013); MacQueen (1967); Comaniciu & Meer (1979); Ng et al.

Original: and deep learning (e.g., Xie et al.
Revised: and deep learning (e.g., Xie et al.

Original: ; Chang et al.
Revised: ; Chang et al.

Original: Dizaji et al.
Revised: Dizaji et al.

Original: ; Yang et al.
Revised: ; Yang et al.

Original: Hsu et al.
Revised: Hsu et al.

Original: (2018; 2019)) clustering methods have been proposed to automatically partition an unlabelled data collection into different classes.
Revised: (2018; 2019)) clustering methods have been proposed to automatically partition an unlabelled data collection into different classes.

Original: However, this task is usually ill-posed as there are multiple, equally valid criteria to partition most datasets.
Revised: However, this task is usually ill-posed as there are multiple, equally valid criteria to partition most datasets.

Original: We address this challenge by learning the appropriate criterion by using a labelled dataset, narrowing down what constitutes a proper class.
Revised: We address this challenge by learning the appropriate criterion by using a labelled dataset, narrowing down what constitutes a proper class.

Original: We call this setting “transfer clustering”.
Revised: We call this setting “transfer clustering”.

Original: To  our knowledge, the work most related to ours is (Han et al., 2019), which also consider discovering new classes as a transfer clustering problem.
Revised: To the best of our knowledge, the work most related to ours are (Hsu et al., 2018; 2019; Han et al., 2019). Han et al. (2019) also consider discovering new classes as a transfer clustering problem.

Original: They  learn a data embedding by using metric learning on the labelled data, and then use thelatter to cluster  the unlabelled data in search for novel classes.
Revised: They ﬁrst learn a data embedding by using metric learning on the labelled data, and then ﬁnetune the embedding and learn the cluster assignments on the unlabelled data.

Original: Other related clustering methods include KCL (Hsu et al., 2018) and MCL (Hsu et al., 2019).
Revised: In (Hsu et al., 2018; 2019), the authors introduce KCL and MCL clustering methods.

Original: Though the design of these two methods is not optimally for our task, they are applicable to it.

Original: In KCL and MCL, a similarity prediction network (SPN) is ﬁrst trained on a labelled dataset.
Revised: In both, a similarity prediction network (SPN), also used in (Hsu et al., 2016), is ﬁrst trained on a labelled dataset.

Original: Afterwards, the pre-trained SPN is used to provide binary pseudo labels for training the main model on an unlabelled dataset.
Revised: Afterwards, the pretrained SPN is used to provide binary pseudo labels for training the main model on an unlabelled dataset.

Original: The overall pipelines of the two methods are similar, but the losses differ: 
Revised: The overall pipelines of the two methods are similar, but the losses differ: KCL uses a

Original: KLC uses a Kullback-Leibler divergence based contrastive loss, and MLC uses the Meta Classiﬁcation Likelihood loss.
Revised: Kullback-Leibler divergence based contrastive loss equivalent to the BCE used in this paper (eq.3), and MCL uses the Meta Classiﬁcation Likelihood loss.

Original: Zero-shot learning (ZSL)
Revised: Zero-shot learning (ZSL)

Original: (Xian et al., 2018; Fu et al., 2018) can also be used to recognize new classes.
Revised: (Xian et al., 2018; Fu et al., 2018) can also be used to recognize new classes.

Original: However, differently from our work, ZSL also requires additional side information (e.g., class attributes) in addition to the raw images.
Revised: However, differently from our work, ZSL also requires additional side information (e.g., class attributes) in addition to the raw images.

Original: Finally, other works (Dean et al., 2013; Yagnik et al., 2011) discuss the application of rank statistics to measuring the similarity of vectors; however, to the best of our knowledge, we are the ﬁrst to apply rank statistics to the task of novel category discovery using deep neural networks.
Revised: Finally, other works (Dean et al., 2013; Yagnik et al., 2011) discuss the application of rank statistics to measuring the similarity of vectors; however, to the best of our knowledge, we are the ﬁrst to apply rank statistics to the task of novel category discovery using deep neural networks.

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: In this paper, we have looked at the problem of discovering new classes in an image collection, leveraging labels available for other, known classes.
Revised: In this paper, we have looked at the problem of discovering new classes in an image collection, leveraging labels available for other, known classes.

Original: We have shown that this task can be addressed very successfully by a few new ideas.
Revised: We have shown that this task can be addressed very successfully by a few new ideas.

Original: First, the use of self-supervised learning for bootstrapping the image representation trades off the representation quality with its generality, and for our problem this leads to a better solution overall.
Revised: First, the use of self-supervised learning for bootstrapping the image representation trades off the representation quality with its generality, and for our problem this leads to a better solution overall.

Original: Second, we have shown that rank statistics are an effective method to compare noisy image descriptors, resulting in robust data clustering.
Revised: Second, we have shown that rank statistics are an effective method to compare noisy image descriptors, resulting in robust data clustering.

Original: Third, we have shown that jointly optimizing both labelled recognition and unlabelled clustering in an incremental learning setup can reinforce the two tasks while avoiding forgetting.
Revised: Third, we have shown that jointly optimizing both labelled recognition and unlabelled clustering in an incremental learning setup can reinforce the two tasks while avoiding forgetting.

Original: In standard benchmarks, the combination of these ideas results in much better performance than existing methods that solve the same task.
Revised: In standard benchmarks, the combination of these ideas results in much better performance than existing methods that solve the same task.

Original: A D ATASET SPLITS
Revised: A D ATASET SPLITS

Original: For CIFAR-10 and SVHN we keep the labels of the ﬁve ﬁrst categories (namely airplane, automobile, bird, cat, deer for CIFAR-10, 0–4 for SVHN) and keep the rest of the data as the unlabelled set.
Revised: For CIFAR-10 and SVHN we keep the labels of the ﬁve ﬁrst categories (namely airplane, automobile, bird, cat, deer for CIFAR-10, 0–4 for SVHN) and keep the rest of the data as the unlabelled set.

Original: For CIFAR-100 we use the ﬁrst 80 categories as labelled data while the rest are used for the unlabelled set.
Revised: For CIFAR-100 we use the ﬁrst 80 categories as labelled data while the rest are used for the unlabelled set.

Original: Following Hsu et al.
Revised: Following Hsu et al.

Original: (2018; 2019), for OmniGlot, each of the 20 alphabets in evaluation set (with 20–47 categories, 659 characters/class) is used as unlabelled data, and all the 30 alphabets in background set are used as labelled set (964 characters/class).
Revised: (2018; 2019), for OmniGlot, each of the 20 alphabets in evaluation set (with 20–47 categories, 659 characters/class) is used as unlabelled data, and all the 30 alphabets in background set are used as labelled set (964 characters/class).

Original: For ImageNet, we follow Hsu et al.
Revised: For ImageNet, we follow Hsu et al.

Original: (2018; 2019) to use the 882/118 classes split proposed in Vinyals et al.
Revised: (2018; 2019) to use the 882/118 classes split proposed in Vinyals et al.

Original: and use the three 30-class subsets sampled from the 118 classes as unlabelled sets.
Revised: and use the three 30-class subsets sampled from the 118 classes as unlabelled sets.

Original: B I MPLEMENTATION DETAILS
Revised: B I MPLEMENTATION DETAILS

Original: In the ﬁrst self-supervised training step, otherwise mentioned, we trained our model with the pretext task of rotation predictions (i.e., a four-class classiﬁcation: 0 ◦ , 90 ◦ , 180 ◦ , and 270 ◦ ) for 200 epochs and a step-wise decaying learning rate starting from 0.1 and divided by 5 at epochs 60, 120, and 160.
Revised: In the ﬁrst self-supervised training step, otherwise mentioned, we trained our model with the pretext task of rotation predictions (i.e., a four-class classiﬁcation: 0 ◦ , 90 ◦ , 180 ◦ , and 270 ◦ ) for 200 epochs and a step-wise decaying learning rate starting from 0.1 and divided by 5 at epochs 60, 120, and 160.

Original: In the second step of our framework (i.e., supervised training using labelled data), we ﬁne-tuned our model on the labelled set for 100 epochs and a step-wise decaying learning rate starting from 0. and halved every 10 epochs.
Revised: In the second step of our framework (i.e., supervised training using labelled data), we ﬁne-tuned our model on the labelled set for 100 epochs and a step-wise decaying learning rate starting from 0. and halved every 10 epochs.

Original: From this step onward we ﬁx the ﬁrst three convolutional blocks of the model, and ﬁne-tuned the last convolutional block together with the linear classiﬁer.
Revised: From this step onward we ﬁx the ﬁrst three convolutional blocks of the model, and ﬁne-tuned the last convolutional block together with the linear classiﬁer.

Original: Finally, in the last joint training step, we ﬁne-tuned our model for 200/100/90 epochs for { CIFAR10, CIFAR-100, SVHN } /OmniGlot/ImageNet, which was randomly sampled from the merged set of both labelled and unlabelled data.
Revised: Finally, in the last joint training step, we ﬁne-tuned our model for 200/100/90 epochs for { CIFAR10, CIFAR-100, SVHN } /OmniGlot/ImageNet, which was randomly sampled from the merged set of both labelled and unlabelled data.

Original: The initial learning rate was set to 0.1 for all datasets, and was decayed with a factor of 10 at the 170th/ { 30th, 60th } epoch for { CIFAR-10, CIFAR100, SVHN } /ImageNet.
Revised: The initial learning rate was set to 0.1 for all datasets, and was decayed with a factor of 10 at the 170th/ { 30th, 60th } epoch for { CIFAR-10, CIFAR100, SVHN } /ImageNet.

Original: The learning rate of 0.01 was kept ﬁxed for OmniGlot.
Revised: The learning rate of 0.01 was kept ﬁxed for OmniGlot.

Original: For the consistency regularization term, we used the ramp-up function as described in section 2.4 with λ = { 5 . 0 , 50 . 0 , 50 . 0 , 100 . 0 , 10 . 0 } , and T = { 50 , 150 , 80 , 1 , 50 } for CIFAR-10, CIFAR-100, SVHN,
Revised: For the consistency regularization term, we used the ramp-up function as described in section 2.4 with λ = { 5 . 0 , 50 . 0 , 50 . 0 , 100 . 0 , 10 . 0 } , and T = { 50 , 150 , 80 , 1 , 50 } for CIFAR-10, CIFAR-100, SVHN,

Original: OmniGlot, and ImageNet respectively.
Revised: OmniGlot, and ImageNet respectively.

Original: In the incremental learning setting, all previous hyper parameters remain the same for our method.
Revised: In the incremental learning setting, all previous hyper parameters remain the same for our method.

Original: We only add a ramp-up on the cross entropy loss on unlabelled data.
Revised: We only add a ramp-up on the cross entropy loss on unlabelled data.

Original: The ramp-up length is the same as the one used for eq.
Revised: The ramp-up length is the same as the one used for eq.

Original: (4) and we use for all experiments a coefﬁcient of 0.05.
Revised: (4) and we use for all experiments a coefﬁcient of 0.05.

Original: For all other methods we trained the classiﬁer for 150 epochs with SGD with momentum and learning rate of 0.1 divided by 10 at epoch 50.
Revised: For all other methods we trained the classiﬁer for 150 epochs with SGD with momentum and learning rate of 0.1 divided by 10 at epoch 50.

Original: We implemented our method using PyTorch 1.1.0 and ran experiments on NVIDIA Tesla M
Revised: We implemented our method using PyTorch 1.1.0 and ran experiments on NVIDIA Tesla M

Original: GPUs.
Revised: GPUs.

Original: Following (Han et al., 2019), our results were averaged over 10 runs for all datasets, except for ImageNet which was averaged over the three 30-class subsets.
Revised: Following (Han et al., 2019), our results were averaged over 10 runs for all datasets, except for ImageNet which was averaged over the three 30-class subsets.

Original: In general, we found the results were stable.
Revised: In general, we found the results were stable.

Original: Our code will be made publicly available upon acceptance.
Revised: Our code will be made publicly available upon acceptance.

Revised: C S TANDARD DEVIATION OF I NCREMENTAL LEARNING EXPERIEMENT

Revised: [Table]

Revised: [Figure]

Revised: CC .

Revised: Performance of our method w.r.t k

Revised: SVHN CIFAR

Revised: CIFAR

Revised: We report results for k = { 1 , 2 , 3 , 5 , 7 , 10 , 15 , 20 , 50 } .

Revised: D I MPACT OF k OVER RESULTS

Revised: We provide an additional study of the evolution of performances of our method with respect to k .

Revised: We results on SVHN/CIFAR10/CIFAR100 in ﬁg. 4.

Revised: We found that k = { 5 , 7 } gave the best results overall.

Revised: We also found that for all values of k except 1 results were in general stable.