A Closer Look at Novel Class Discovery from the Labeled Set
Abstract
Novel class discovery (NCD) is to infer novel categories in an unlabeled set using prior knowledge of a labeled set comprising diverse but related classes.
Existing research focuses on using the labeled set methodologically and little on analyzing it.
In this study, we closer look at NCD from the labeled set and focus on two questions: (i) Given an unlabeled set, what labeled set best supports novel class discovery?
(ii) A fundamental premise of NCD is that the labeled set must be related to the unlabeled set, but how can we measure this relation?
For (i), we propose and substantiate the hypothesis that NCD could benefit from a labeled set with high semantic similarity to the unlabeled set.
Using ImageNet’s hierarchical class structure, we create a large-scale benchmark with variable semantic similarity across labeled/unlabeled datasets.
In contrast, existing NCD benchmarks ignore the semantic relation.
For (ii), we introduce a mathematical definition for quantifying the semantic similarity between labeled and unlabeled sets.
We utilize this metric to validate our established benchmark and demonstrate it highly corresponds with NCD performance.
Furthermore, without quantitative analysis, previous works commonly believe that label information is always beneficial.
However, counterintuitively, our experimental results show that using labels may lead to sub-optimal outcomes in low-similarity settings.
Introduction
Deep models are capable of identifying and clustering classes that are present in the training set (i.e., known/seen classes), matching or surpassing human performance.
However, they lack reliable extrapolation capacity when confronted with novel classes while humans can easily recognize the unseen categories.
This encouraged researchers to establish a challenge termed novel class discovery (NCD) [10, 2, 9, 22], to identify new classes in an unlabeled dataset by utilizing information from a labeled set containing similar but disjoint classes.
Currently, most NCD research takes place at the method level, focusing on how to better utilize the labeled set.
Though the labeled set is essential, there is less in-depth analysis of the labeled set itself.
This lack of understanding about a crucial aspect of NCD illustrates the necessity to explore it from the labeled set’s perspective.
Thus, our paper concentrates on two core questions: First, given a specific unlabeled set, what kind of labeled set can best support novel class discovery?
Second, an essential premise of NCD is that the labeled set should be related to the unlabeled set, but how can we measure this relation?
Additionally, based on the preceding questions, we also give insights on the importance of labeled information in NCD.
Regarding the first question, we intuitively expect that labeled sets with higher semantic similarity can provide more beneficial knowledge while the number of categories and pictures is fixed.
In contrast, existing works solely use the number of labeled/unlabeled classes and images to determine NCD difficulty, e.g., [5, 2], and disregard semantic similarity.
We first verify our assumption on multiple pairs of labeled and unlabeled sets with varying semantic similarity and under multiple baselines [10, 9, 5, 16].
Then, we establish a new benchmark with multiple semantic similarity levels using
ImageNet’s hierarchical semantic information.
Section 2 provides details.
Second, an essential premise of NCD is that leveraging the information of the disjoint but related labeled set improves performance on the unlabeled data.
A prior work [2] points out that NCD is theoretically solvable when labeled set and unlabeled set share high-level semantic features yet without proposing any quantitative analysis.
This inspires the following questions: How closely related do the sets need to be for NCD to work?
How can we measure the semantic relatedness between labeled and unlabeled sets?
Motivated by these questions, we propose a semantic similarity metric, called transfer leakage .
Specifically, transfer leakage quantifies how much information we can leverage from the labeled dataset to help improve the performance of the unlabeled dataset, more detail in Section 3.
Furthermore, we observe that labeled information may lead to sub-optimal results, contrary to the commonly held belief that labeled information is always beneficial for NCD tasks.
However, it is hard to decide whether to use labeled supervised knowledge or self-supervised knowledge without labels.
Thus, we provide two concrete solutions.
(i) pseudo transfer leakage , a practical reference for what sort of data we intend to employ.
(ii) A straightforward method, which smoothly combines supervised and self-supervised knowledge from the labeled set, and achieve 3% and 5% improvement in both CIFAR100 and ImageNet compared to SOTA, further information in Section 4.
We summarize our contributions as follows: (i)
We establish a comprehensive and large benchmark with varying degrees of difficulty on ImageNet and thoroughly justify the assumption that semantic similarity is indeed a significant factor influencing NCD performance.
(ii)
We introduce a mathematical definition for evaluating the semantic similarity between labeled and unlabeled sets and validate it under CIFAR100 and ImageNet.
(iii)
We observe counterintuitive results - labeled information may lead to suboptimal performance and propose two practical applications, which achieve 3% and 5% improvement in both CIFAR100 and ImageNet compared to SOTA.
Assumption and Proposed Benchmarks
In this section, we address the first question: given a specific unlabeled set, what kind of labeled set can best support novel class discovery?
We first assume that higher semantic similarity labeled sets can provide greater help compared to less similar labeled sets when the number of categories and images in the labeled sets are fixed.
However, existing benchmarks were created based on the number of categories(e.g., [5]) and images(e.g.
,[2]) without considering semantic similarity between the two sets.
Thus, we propose a new benchmark based on the ENTITY-30 task[18] including three different semantic similarity levels (high, medium and low) by leveraging the underlying hierarchy, which contains 240 ImageNet classes in total, with 30 superclasses and 8 subclasses for each superclasses.
For our benchmark, we use these classes to create NCD tasks with 90 labeled and 30 unlabeled classes each.
Lastly, our hypothesis is verified on our benchmark (Table 1) and CIFAR100 (Table 3).
The results demonstrate that the most similar labeled set achieves the highest performance, followed by the medium and the least similar set.
Further details and experimental setup are in the Appendix A.
Quantifying Semantic Similarity 3.1 NCD Framework
We denote ( X l , Y l ) and ( X u , Y u ) as random samples under the labeled/unlabeled probability measures P X ,Y and Q X ,Y , respectively.
X l ∈ X l ⊂ R d and X u ∈ X u ⊂ R
d are the labeled/unlabeled feature vectors, Y l ∈ C l and Y u ∈ C u are the true labels of labeled/unlabeled data, where C l and C u
[Table]
[Table]
are the label sets under the labeled and unlabeled probability measures P X ,Y and Q X ,Y , respectively.
Given a labeled set L n =
( X l,i , Y l,i ) ni
=1 independently drawn from the labeled probability measure P
X ,Y , and an unlabeled dataset U m =
( X u,i ) mi
=1 independently drawn from the unlabeled probability measure Q X u , our primary goal is to predict Y u,i given X u,i , where Y u,i is the label of the i -th unlabeled sample X u,i .
Definition 1 (Novel class discovery) Let P X l ,Y l be a labeled probability measure on X l × C l , and Q
X u ,Y u be an unlabeled probability measure on X u × C u , with C u ∩ C l = ∅ .
Given a labeled dataset
L n sampled from P X l ,Y l and an unlabeled dataset U m sampled from Q X u , novel class discovery aims to predict the labels of the unlabeled dataset based on L n and U m .
3.2 Transfer Leakage
We begin with introducing Maximum Mean Discrepancy (MMD) [7], which is used to measure the discrepancy of two distributions.
For example, the discrepancy of two random variables Z ∼ P Z and Z ′ ∼ P Z ′ is defined as: MMD H (cid:0) P Z , P Z ′ (cid:1) :
= sup ∥ h
∥ H ≤ 1 (cid:16) E (cid:0) h ( Z ) (cid:1) − E (cid:0) h
( Z ′ )
(cid:1)(cid:17) where H is a class of functions
h : X u → R , which is specified as a reproducing kernel Hilbert Space (RKHS) associated with a continuous kernel function K ( · , · ) .
In NCD, the unlabeled dataset utilizes the conditional probability P Y l | X l (usually presented by a pretrained neural network) from a labeled dataset.
For example, if the distributions of P Y l | X l = X u under Y u
= c
and Y u
= c ′ are significantly different, then its overall distribution discrepancy is large, yielding that more information can be leveraged in NCD.
On this ground, we use MMD to quantify the discrepancy of the labeled probability measure P Y l | X l on X u under the unlabeled probability measure Q , namely transfer leakage .
Definition 2 (Transfer leakage)
The transfer leakage of NCD prediction under Q based on the labeled conditional probability P Y l | X l is
[Equation]
where ( X u , Y u ) , ( X ′ u , Y ′ u ) ∼ Q are independent copies, the expectation E Q is taken with respect to Y u and Y ′ u under Q , and p ( x ) is the conditional probability under P Y l | X l on an unlabeled data
X u = x , which is defined as p ( x
[Equation]
To summarize, transfer leakage measures the overall discrepancy of p ( X u ) under different new classes of the unlabeled measure Q , which indicates the informative leakage from P to Q .
Next, we give a finite sample estimate of transfer leakage .
Given an estimated probability (cid:98) P Y l | X l and an evaluation dataset ( x u,i , y u,i ) mi =1 under Q , we assess x u,i on (cid:98) P Y l | X l as (cid:98) p ( x u,i ) =
(cid:0)(cid:98) P (cid:0)
Y l = c | X l = x u,i (cid:1)(cid:1) ⊺ c ∈C u , then empirical transfer leakage is computed as:
[Equation]
where the equality follows from the fact that MMD 2 H (cid:0) Q p
[Equation] ) | Y u = c , Q p ( X ′ u ) | Y ′ u
= c (cid:1) = 0 .
I u,c
= { 1 ≤ i ≤ m : y u,i = c } is the index set of unlabeled data with y u,i
= c , and (cid:92) MMD 2 H is defined in the Appendix.
We use the proposed (cid:92) T-Leak to quantify the difficulty of NCD in various combination of labeled and unlabeled sets.
It is worth noting that the proposed transfer leakage and its empirical evaluation depend only on the labeled/unlabeled datasets, and it remains the same, no matter what
NCD method we use.
In addition, we provide pseudo transfer leakage (Appendix), a practical evaluation of the similarity between the classified and unlabeled sets.
transfer leakage utilizes the true label Y u , but the pseudo transfer leakage utilizes the pseudo label obtained from clustering methods (e.g., k -means) on the representations.
[Table]
Supervised Knowledge may be Harmful
The motivation behind NCD is that supervised knowledge from labeled data can enhance unlabeled data clustering.
Counterintuitively, we observe that supervised information from a labeled set may result in suboptimal outcomes as compared to using exclusively self-supervised knowledge.
Experiments and Results To investigate, we conduct experiments in the following settings: (1)
Using the unlabeled set, X u ; (2) Using the unlabeled set and the labeled set’s images without labels,
X u + X l ; (Even without labels, self-supervised learning can extract the knowledge from labeled set’s image.)
(3) Using the unlabeled set and the whole labeled set, X u + ( X l , Y l ) , (i.e., standard NCD).
As suggested in Table 2, NCD performance is consistently improved by incorporating more images (without labels) from a labeled set, around 10% on CIFAR100 and 6%-18% in our benchmark.
By comparing (2) and (3), we can isolate the impact of the labels.
Unexpectedly, on CIFAR100-50 and
ImageNet with low semantic similarity, (3) performs around 2 - 8% worse than (2), yielding that “low-quality" supervised information may hurt NCD performance.
Practical Applications As shown above, supervised knowledge from the labeled set may cause damage, but it’s difficult to determine whether to utilize it or self-supervised knowledge.
Therefore, we offer two concrete solutions, a practical metric (i.e., pseudo transfer leakage ) and a straightforward method.
The proposed pseudo transfer leakage is a practical reference to infer what kind of data we want to employ in NCD, images-only information or image-label pairs.
Table 7 indicates that pseudo transfer leakage is consistent with various settings’ accuracy.
E.g., in L 2 - U 1 , the pseudo transfer leakage computed on the self-supervised model is larger than on the supervised one, which is consistent with accuracy, where X u + X l outperforms
X u
+ ( X l , Y l ) .
We also present an approach that combines supervised and self-supervised knowledge.
We combine the labeled set’s ground truth labels y l GT with self-supervised pseudo labels y l PL .
αy l GT + (1 − α )
y l PL is the overall classification objective, where α ∈ [0 , 1] is the supervised component weight.
This strategy has the same aim as UNO [5] for alpha = 1 , but uses self-supervised pretraining instead of supervised.
As shown in Figure 3, our proposed method improves CIFAR100 and ImageNet by 3% and 5%, respectively, compared to UNO.
Our method delivers significant improvements for low semantic similarity cases and competitive performances for high similarity cases.
Further information in Appendix C.2.
Conclusion
We first offer a comprehensive ImageNet-based benchmark with varying levels of semantic similarity and show that semantic similarity affects NCD performance.
Second, we present transfer leakage , a semantic similarity metric.
Furthermore, we find that in low semantic similarity situations, labeled information may lead to inferior performance.
We propose two practical applications based on these findings: (i) Using transfer leakage to determine what data to use.
(ii) a straightforward approach that improves CIFAR100 and ImageNet by 3-5%.
A Details of SectionA.1 Assumption
Existing benchmarks consider the difficulty of NCD in terms of the labeled set from two aspects: (1) the number of categories, e.g. [5] propose a more challenging benchmark called CIFAR100-50 (i.e., 50/50 classes for unlabeled/labeled set), compared to the commonly used CIFAR100-20 (i.e., 20/ classes for unlabeled/labeled set).
The number of images in each category, [2] propose to use less images for each labeled’s class.
However, in addition to the number of categories and images, another significant factor is the semantic similarity between the two sets.
As mentioned in [2], NCD is theoretically solvable when labeled and unlabeled sets share high-level semantic features.
Based on this, we conduct a further investigation with the assumption that more similar labeled sets (when the number of categories and images are fixed) can lead to better performance.
Intuitively, according to Figure 1, despite the fact that the labeled (e.g., pineapple, strawberry) and unlabeled (e.g., pear, jackfruit) classes are disjoint, if they derive from the same superclass (i.e., fruit), they have a higher degree of semantic similarity.
Conversely, when labeled (e.g., rose, lotus) and unlabeled (e.g., pear, jackfruit) classes are derived from distinct superclasses (i.e., labeled classes from flower while unlabeled classes from fruit), they are further apart semantically.
Consequently, we construct various semantic similarity labeled/unlabeled settings based on a hierarchical class structure and evaluate our assumption on
CIFAR100 and ImageNet.
A.2 Benchmark
Existing benchmarks in the field were created without regard to the semantic similarity between labeled and unlabeled set.
Most works follow the standard splits introduced in [10].
In CIFAR10 [15], the labeled set is made up of the first five classes in alphabetical order, and the unlabeled set of the remaining five.
A similar approach was taken with the commonly used CIFAR100-20 and CIFAR10050 benchmarks.
A benchmark based on ImageNet [4] has one labeled set, with 882 classes and three unlabeled sets.
Each of these unlabeled sets contains 30 classes, which were randomly selected from the remaining non-labeled classes [19, 12, 14, 10].
To address this limitation and allow for an evaluation of our assumptions, we propose a new benchmark based on ImageNet including three different semantic similarity levels (high, medium and low).
As mentioned in Section A.1, we separate labeled and unlabeled classes by leveraging ImageNet’s underlying hierarchy.
While ImageNet is based on the WordNet hierarchy [17], it is not well-suited for this purpose as discussed by [18].
To address these issues, they propose a modified hierarchy and define multiple hierarchical classification tasks based on it.
While originally defined to measure the impact of subpopulation shift, they can also be used to define NCD tasks.
Our proposed benchmark is based on the ENTITY-30 task, which contains 240 ImageNet classes in total, with 30 superclasses and 8 subclasses for each superclasses.
For example, as shown in Figure 1, we define three labeled sets L 1 , L 2 and L 1 . 5 and two unlabeled sets U 1 and U 2 .
The sets L 1 and
U 1 are selected from the first 15 superclasses, with 6 subclasses of each superclass assigned to Land the other 2 assigned to U 1 .
The sets L 2 and U 2 are created from the second 15 superclasses in a similar fashion.
Finally, L 1 . 5 is created by taking half the classes from L 1 and half of the classes from L 2 .
Therefore, ( U 1 , L 1 )/
( U 2 , L 2 ) are highly related semantically, ( U 1 , L 2 )/
( U 2 , L 1 ) belong to the low semantic cases and ( U 1 , L 1 . 5 )/
( U 2 , L 1 . 5 ) are the medium cases.
Additionally, we also create four data settings on CIFAR100, with two high semantic cases and two low semantic cases by leveraging CIFAR100 hierarchical class structure.
Each case has 40 labeled classes and 10 unlabeled classes.
A full list of the labeled and unlabeled sets with their respective superclasses and subclasses can be found in Appendix G.
This benchmark setup allows us to systematically investigate the influence of the labeled set on a large benchmark dataset.
By keeping the unlabeled set constant and varying the used labeled set, we can isolate the influence of semantic similarity on NCD performance.
[Figure]
Unlabeled set U 1 and labeled set L 1 are from the same superclass (fruit), whereas unlabeled set Uand labeled set L 2 belong to another superclass (flower).
Labeled set L 1 . 5 is composed of half of Land half of L 2 .
If both the labeled and unlabeled classes are derived from the same superclass, i.e.
( U 1 , L 1 ) and ( U 2 , L 2 ), we consider them a high semantic similarity split.
In contrast, ( U 1 , L 2 ) and ( U 2 , L 1 ) are low semantic similarity splits, since the labeled and unlabeled classes are derived from distinct superclasses.
In addition, we consider ( U 1 , L 1 . 5 ) and ( U 2 , L 1 . 5 ) to have medium semantic similarity because half of L 1 . 5 share the same superclass as U 1 .
A.3 Experimental Setup and Results
To verify our assumption, we conduct experiments on 4 competitive baselines, including K-means [16], DTC [10], RS [9], NCL [22] and UNO [5].
We follow the baselines regarding hyperparameters and implementation details.
Results on CIFAR
In Table 3, U 1 and U 2 represent the unlabeled sets, while L 1 and Lrepresent the labeled sets.
U 1 / U 2 and L 1 / L 2 share the same super classes, while U 1 / U 2 and L 2 / Lbelong to different super classes.
We evaluate 4 different labeled/unlabeled settings in CIFAR100, with 2 high semantic cases (i.e., ( U 1 , L 1 ) and ( U 2 , L 2 )) and low semantic cases (i.e., ( U 1 , L 2 ) and ( U 2 , L 1 )).
The gap between the high-similarity and the low-similarity settings is larger than 20% for
K-means, and reaches up to 12% for more advanced methods.
The strong results of UNO across all splits show that a more difficult benchmark is needed to obtain clear results for future methods.
Results on our proposed benchmark Similarly, in Table 1, the most similar labeled set generally obtains the best performance, followed by the medium and the least similar one.
Under the unlabeled set U 1 , L 1 achieves the highest accuracy, with around 2-17% improvement compared to L 2 , and around 2-11% improvement compared to L 1 . 5 .
For the unlabeled set U 2 , L 2 is the most similar set and obtains 8-14% improvement compared to L 1 , and around 5-14% improvement compared to L 1 . 5 .
[Table]
B Details of SectionB.1 The Detailed Derivation Process of Transfer Leakage
To summarize, transfer leakage measures the overall discrepancy of p ( X u ) under different new classes of the unlabeled measure Q , which indicates the informative leakage from P to Q .
Definition 3 (Transfer leakage)
The transfer leakage of NCD prediction under Q based on the labeled conditional probability P Y l | X l is
[Equation]
where ( X u , Y u ) , ( X ′ u , Y ′ u ) ∼ Q are independent copies, the expectation E Q is taken with respect to Y u and Y ′ u under Q , and p ( x ) is the conditional probability under P Y l | X l on an unlabeled data
X u = x , which is defined as p ( x
[Equation]
= (cid:0)
[Equation]
(cid:0)
[Equation]
Next, we give a finite sample estimate of transfer leakage .
To proceed, we first rewrite transfer leakage as follows.
[Equation]
where the equality follows from the fact that MMD 2 H (cid:0) Q p ( X u ) | Y u
= c , Q p
[Equation]
X
[Equation]
| Y
[Equation]
u = c (cid:1) = 0 .
Given an estimated probability (cid:98) P Y l | X l and an evaluation dataset ( x u,i , y u,i ) mi =1 under Q , we assess x u,i on (cid:98) P Y l | X l as (cid:98)
[Equation]
u , then the empirical transfer leakage is computed as:
[Equation]
where I u,c = { 1 ≤ i ≤ m : y u,i = c } is the index set of unlabeled data with y u,i =
c
[Equation]
and (cid:92) MMD 2 H is defined as:
[Equation]
B.2 Pseudo Transfer Leakage
In practice, the ground-truth labels of the unlabeled set are difficult to acquire.
Therefore, we utilize pseudo labels derived from clustering algorithms, e.g., k -means, rather than true labels in computing transfer leakage, and we named it as pseudo transfer leakage
Definition 4 (Pseudo Transfer Leakage)
[Equation]
where (cid:101) I u,c = { 1 ≤ i ≤ m : (cid:101)
y u,i =
c } is the index set of unlabeled data with (cid:101)
y u,i =
c , (cid:101) y u,i is provided based on k-means on their representations (cid:98) s ( x u,i ) , and (cid:98) s ( x u,i ) is the representation estimated from a supervised model or a self-supervised model.
(cid:92) MMD 2 H is defined similarly as in
Section 3.
In addition, we provide pseudo transfer leakage , a practical evaluation of the similarity between the classified and unlabeled sets.
transfer leakage utilizes the true label Y u , but the pseudo transfer leakage utilizes the pseudo label obtained from k -means on the representations.
Furthermore, pseudo transfer leakage could be a practical reference on whether to images-only or image-label pairs information from the labeled set, more detail in Section 4 and Appendix C.2.
In Section B.4, we use the proposed (cid:92) T-Leak to quantify the difficulty of NCD in various combination of labeled and unlabeled sets.
It is worth noting that the proposed transfer leakage and its empirical evaluation in (5) depend only on the labeled/unlabeled datasets, and it remains the same, no matter what NCD method we use.
B.3 The Lower and Upper Bounds of Transfer Leakage
Lemma 1 shows the lower and upper bounds of transfer leakage , and provide a theoretical justification that it is a effective quantity to measure the similarity between labeled and unlabeled datasets.
Lemma 1 Let κ := max c ∈C u E Q (cid:0)(cid:112) K ( p ( X u ) , p ( X u ))
| Y u
= c (cid:1) < ∞ , then 0 ≤ T-Leak ( Q , P ) ≤ 4 κ 2 .
Moreover, T-Leak ( Q , P ) =
if and only if Y u is independent with p ( X u ) , that is, for any c ∈ C u :
[Equation]
7) yielding that p ( X u ) is useless in NCD on Q .
Note that κ can be explicitly computed for many common used kernels, for example, κ = 1 for a
Gaussian or Laplacian kernel.
From Lemma 1, T-Leak ( Q , P ) = 0 is equivalent to Y u is independent with p ( X u ) , which matches our intuition of no leakage.
Alternatively, if Y u is dependent with p ( X u ) , we justifiably believe that the information of Y l | X l can be used to facilitate NCD, Lemma 1 tells that
T-Leak ( Q , P ) > 0 in this case.
Therefore, Lemma 1 reasonably suggests that the proposed transfer leakage is an effective metric to detect if the labeling information in P is useful to NCD on Q
[Equation]
Proof of Lemma 1.
We first show the upper bound of the transfer leakage.
According to Lemma 3 in [7]
[Equation]
we have
[Equation]
where µ Q s ( X u ) | Yu :
= E Q (cid:0) K ( s ( X u ) , · ) | Y u (cid:1) is the kernel mean embedding of the measure Q s ( X u ) | Y u [7], the second inequality follows from the triangle inequality in the Hilbert space, the fourth equality follows from the fact that E Q is a linear operator, the second last inequality follows from the CauchySchwarz inequality, and the last equality follows the reproducing property of K ( · , · ) .
Next, we show the if and only if condition for T-Leak ( Q , P ) = 0 .
Assume that Q ( Y u = c ) > 0 for all c ∈ C u .
According to Theorem 5 in [7], we have
[Equation]
Note that 1 = (cid:88) c ∈C u
Q
[Equation]
|
[Equation]
( x
[Equation]
= (cid:88) c ∈C u
Q (
[Equation]
( x
[Equation]
| Y u = c ) Q ( Y u = c )
Q ( s ( x
[Equation]
= (cid:88) c ∈C u q 0 ( x ) Q
( Y u = c )
Q ( s ( x ))
=
q 0 ( x )
Q ( s ( x )) , yielding that Q (cid:0) s ( x ) | Y u
= c (cid:1) =
Q ( s ( x )) , for c ∈ C u , x ∈ X u .
This is equivalent to,
[Equation]
This completes the proof.
□
[Figure]
Each line stands for one unlabeled set from our proposed ImageNet-based benchmark, and each point on the line for one / labeled unlabeled split.
For each split, we measure the transfer leakage and the clustering accuracy obtained using UNO, NCL and K-means.
As expected, there is a positive correlation between semantic similarity and transfer leakage as well as between transfer leakage and accuracy.
Experiments
[Figure]
B.4 Experiments and Results
Experimental Setup We evaluate transfer leakage on CIFAR100 and our proposed ImageNet-based benchmark under different semantic similarity settings.
This includes four settings on CIFAR100, consisting of two high-similarity and two low-similarity settings, as well as six settings on ImageNet, two each for high, medium and low semantic similarity.
We employ ResNet18 [11] as the backbone for both datasets following [10, 9, 5].
We first apply fully supervised learning to the labeled data for each data set to obtain the pretrained model.
Then, we feed the unlabeled data to the pretrained model to obtain its representation.
Lastly, we calculate the transfer leakage and pseudo transfer leakage based on the pretrained model and the unlabeled samples’ representation.
Hyperparameters For the first step, batch size is set to 512 for both datasets We use an SGD optimizer with momentum 0.9, and weight decay 1e-4.
The learning rate is governed by a cosine annealing learning rate schedule with a base learning rate of 0.1, a linear warmup of 10 epochs, and a minimum learning rate of 0.001.
We pretrain the backbone for 200/100 epochs for CIFAR100/ImageNet.
Results Figure 2 shows the results on ImageNet.
Here, results that share the same color are using the same unlabeled set.
These experiments confirm the validity of transfer leakage as a metric to quantify semantic similarity and the difficulty of a particular NCD problem.
As anticipated, splits that have a higher semantic similarity also have a higher transfer leakage .
Additionally, there is positive correlation between transfer leakage and the clustering accuracy achieved.
[Table]
C Details of SectionC.1 Counterintuitive Observations
The motivation behind NCD is that prior supervised knowledge from labeled data can help improve the clustering of the unlabeled set.
However, we have counterintuitive results: supervised information from a labeled set may result in suboptimal outcomes as compared to using exclusively self-supervised knowledge.
To investigate
[Table]
we conduct experiments in the following settings:
(1) Using the unlabeled set, X u (2) Using the unlabeled set and the labeled set’s images without labels, X u + X l (3) Using the unlabeled set and the whole labeled set, X u + ( X l , Y l ) , (i.e., standard NCD).
Specifically, for (1), NCD is degenerated to unsupervised learning (i
[Table]
e., clustering on X u ).
For (2), even though we do not use the labels, we can still try to extract the knowledge of the labeled set via
-supervised learning.
By comparing (1) and (3), we can estimate the total performance gain caused by adding the labeled set.
The comparison between (1) and (2) as well as (2) and (3) allows us to further disentangle the performance gain according to the components of the labeled set.
[Table] [Table]
3) X u + ( X l , Y l ) 62.2 ± 0.
83.9 ± 0.
81.0 ± 0.
77.2 ± 0.
[Table]
Experimental Setup We conduct experiments on UNO [5], which is to the best of our knowledge the current state-of-the-art method in NCD.
To perform experiments (1) and (2), we make adjustments on the framework of UNO, enabling it to run fully self-supervised.
This is done by replacing the labeled set’s ground truth labels y l GT with self-supervised pseudo labels y l PL , which are obtained by applying the Sinkhorn-Knopp algorithm [3].
The standard UNO method conducts NCD in a two-step approach.
In the first step, it applies a supervised pretraining on the labeled data only.
The pretrained model is then used as an initialization for the second step, in which the model is trained jointly on both labeled and unlabeled data using one labeled head and multiple unlabeled heads.
To adapt UNO to the fully unsupervised setting in (1), we need to remove all parts that utilize the labeled data.
Therefore, in the first step, we replace the supervised pretraining by a self-supervised one, which is trained only on the unlabeled data.
For the second step, we simply remove the labeled head, thus the method is degenerated to a clustering approach based solely on the pseudo-labels generated by the Sinkhorn-Knopp algorithm.
For setting (2), we apply the self-supervised pretraining based on both unlabeled and labeled images to obtain the pretrained model in the first step.
In the second step, we replace the ground-truth labels for the known classes with pseudo-labels generated by the Sinkhorn-Knopp algorithm based on the logits of these classes.
Taken together, the updated setup utilizes the labeled images, but not their labels.
Hyperparameters We conduct our experiments on CIFAR100 as well as our proposed ImageNetbased benchmark.
All settings and hyperparameters are kept as close as possible as to the original baselines, including the choice of ResNet18 as the model architecture.
We use SWAV [1] as selfsupervised pretraining for all experiments.
The pretraining is done using the small batch size configuration of the method, which uses a batch size of 256 and a queue size of 3840.
The training is run for 800 epochs, with the queue being enabled at 60 epochs for our ImageNet-based benchmark and 100 epochs for CIFAR100.
To ensure a fair comparison with the standard NCD setting, the same data augmentations were used.
In the second step of UNO, we train the methods for 500 epochs on
CIFAR100 and 100 epochs for each setting on our benchmark.
The experiments are replicatedtimes on CIFAR100 and 5 times on the developed benchmark, and the averaged performances and their corresponding standard errors are summarized in Table 5.
Results As suggested in Table 5 and Table 6, NCD performance is consistently improved by incorporating more images (without labels) from a labeled set, the percentages of improvements in terms of accuracy are around 10% on CIFAR100 (comparing (1) and (2)).
For our benchmark, the setting (2) obtains an improvement about 6 - 10% over (1) and the increase is more obvious in the lower semantic similarity cases.
Similarly, by comparing (2) and (3), we can isolate the impact of the labels.
For this case, the largest percentage of improvement, about 4 - 7%, is obtained in the high-similarity setting, followed by the the medium-similarity setting with 1 - 3% improvements.
Interestingly, on CIFAR100-50 and ImageNet with low semantic similarity, we unexpectedly observe that (2) performs around 2 - 8% better than (3), yielding that “low-quality" supervised information may hurt NCD performance.
C. Practical Applications
As shown in Table 5 and Table 6, even though we find that supervised knowledge from the labeled set may cause harm rather than gain, it is still difficult to decide whether to utilize supervised knowledge with labeled data or just pure self-supervised knowledge without labels.
Therefore, we offer two concrete solutions, a practical metric (i.e., pseudo transfer leakage ) and a straightforward method.
The proposed pseudo transfer leakage is a practical reference to infer what sort of data we want to use in NCD, images-only information, X u + X l or the image-label pairs, X u + ( X l , Y l ) from the labeled set.
In Table 7, we compute pseudo transfer leakage via a supervised model and a self-supervised model based on pseudo labels.
As suggested„ the pseudo transfer leakage is consistent with the accuracy based on various datasets.
For example, in L 1 − U 1 , the pseudo transfer leakage computed on the supervised model is larger than the one computed in the self-supervised model, which is consistent with accuracy, the supervised method outperforms the self-supervised one.
Reversely, for
L 2 - U 1 , L 1 - U 2 and L 1 .
5 − U 1 , the pseudo transfer leakage computed on the self-supervised model is larger than the one computed in the supervised model, which is again consistent with their relative performance.
For L 2 − U 2 and L 1 .
5 − U 1 , performance in the two settings is within error margins.
Also, we propose a straightforward method, which smoothly combines supervised knowledge and self-supervised knowledge.
Concretely, instead of using either the labeled set’s ground truth labels y l GT or the self-supervised pseudo labels y l PL , we use a linear combination of the two.
The overall classification target is αy l GT + (1 − α )
y l PL , where α ∈ [0 , 1] is the weight of the supervised component.
This means that for α = 1 , this approach has the same target as UNO [5], and differs only in the used pretraining, which is self-supervised as opposed to supervised in UNO.
As indicated in
Figure 3, our proposed method achieves 3% and 5% improvement in both CIFAR100 and ImageNet ( L 2 - U 1 ), respectively compared to UNO, full results can be found in Table 8.
Our method delivers significant improvements for low semantic similarity cases and competitive performances for high semantic cases.
[Table]
[Table] [Table]
D Notations
To proceed
[Table]
we summarize all notations used in the paper in Table
[Table] [Table]
E Related Work
Novel class discovery (NCD) is a relatively new problem proposed in recent years, aiming to discover novel classes (i
[Table]
e., assign them to several clusters) by making use of similar but different known classes.
Compared with unsupervised learning, NCD also requires labeled known-class data to help cluster novel-class data.
NCD is first formalized in DTC [10]
[Table]
but the study of NCD can be dated back to earlier works, such as KCL [13] and MCL [14].
Both of these methods are designed for general task transfer learning
[Table]
and maintain two models trained with labeled data and unlabeled data respectively.
In contrast, DTC first learns a data embedding on the labeled data with metric learning, then employs a deep embedded clustering method based on [20] to cluster the novel-class data.
Later works further deviate from this approach.
Both RS [9, 8] and [21] use self-supervised learning to boost feature extraction and use the learned features to obtain pairwise similarity estimates.
Additionally, [21] improves on RS by using information from both local and global views, as well as mutual knowledge distillation to promote information exchange and agreement.
NCL [22] extracts and aggregates the pairwise pseudo-labels for the unlabeled data with contrastive learning and generates hard negatives by mixing the labeled and unlabeled data in the feature space.
This idea of mixing labeled and unlabeled data is also used in OpenMix [23], which mixes known-class and novel-class data to learn a joint label distribution.
The current state-of-the-art, UNO [5], combines pseudo-labels with ground-truth labels in a unified objective function that enables better use of synergies between labeled and unlabeled data without requiring self-supervised pretraining.
Additionally, there are a few theatrical works.
Meta discovery [2] indicates that NCD is theoretically solvable if known and unknown classes share high-level semantic features and propose a solution that links NCD to meta-learning.
OSLS [6] estimates the target label distribution, including the novel class and learn a target classifier.
F Discussion
The key assumption of novel class discovery is that the knowledge contained in the labeled set can help improve the clustering of the unlabeled set.
Yet, what’s the ‘dark knowledge’ transferred from the labeled set to the unlabeled set is still a mystery.
Therefore, we conduct preliminary experiments to disentangle the impact of the different components of the labeled set, i.e., the images-only information and the image-label pairs.
The results indicate that NCD performance is consistently improved by incorporating more images (without labels) from a labeled set while the supervised knowledge is not always beneficial.
Supervised knowledge (obtained from X l , Y l ) can provide two types of information: classification rule and robustness.
However, self-supervised information from X l is primarily responsible for enhancing model robustness.
In cases of high semantic similarity, the labeled and unlabeled classification rules are more similar than in cases of low semantic similarity.
Thus, supervised knowledge is advantageous in high similarity cases but potentially harmful in low similarity situations while self-supervised knowledge is helpful for both cases.
G Detailed Benchmark Splits
[Table]
balance beam
[Table]
rugby ball
[Table]
ski
[Table]
horizontal bar
[Table]
carnivore otterhound
[Table]
-coated
[Table]
[Table]
Shih
[Table]
they share they
[Table]
dowitcher, goose, albatross, limpkin, white stork, red-backed sandpiper drake, crane ungulate hippopotamus, hog, llama, hartebeest, ox, gazelle warthog, zebra passerine house finch, magpie, goldfinch, indigo bunting, chickadee, brambling bulbul, water ouzel
[Table]
squirrel
[Table]
trees oak_tree
[Table]
palm_tree
[Table]
pine_
[Table]
willow_tree
[Table]
vehiclesbus, motorcycle
[Table]
pickup_truck, train bicycle vehiclesrocket, streetcar, tank, tractor
[Table]