Learning to Discover and Detect Objects
Anonymous Author(s)
AffiliationAddress email
Abstract
We tackle the problem of novel class discovery, detection, and localization (NCDL).
In this setting, we assume a source dataset with labels for objects of commonlyobserved classes.
Instances of other classes need to be discovered, classified, andlocalized automatically based on visual similarity, without human supervision.
Tothis end, we propose a two-stage object detection network Region-based NCDL( RNCDL ), that uses a region proposal network to localize potential objects and classify them.
We train our network to classify each proposal, either as one of the knownclasses, seen in the source dataset, or one of the extended set of novel classes with aconstraint that the distribution of class assignments should follow natural long-taildistributions common in the real open-world.
By training our detection networkwith this objective in an end-to-end manner, it learns to classify all region proposalsfor a large variety of classes, including those that are not part of the labeled objectclass vocabulary.
Our experiments conducted using COCO and LVIS datasets revealthat our method is significantly more effective compared to multi-stage pipelinesthat rely on traditional clustering algorithms or self-supervised contrastive learning methods and operate on pre-extracted crops.
Beyond that, we demonstratethe generality of our approach by applying our method to a large-scale Visual
Genome dataset, where our network successfully learns to detect various semanticclasses without explicit supervision.
Code is available at: https://github.com/5da15f23f9/learning-to-discover-and-detect-objects .
Introduction
We tackle novel class discovery and localization in unlabeled datasets, a long-standing problemin computer vision [55, 59, 41, 42, 54].
As we are approaching the limits of well-understood andsuccessful supervised training of object detectors [22, 21, 53, 27], this is becoming an increasinglymore important problem, with the task of labeling novel and rare classes that appear in the long tailof object class distribution becoming prohibitively expensive [24]
Object class discovery is a difficult problem, as there may be many possible equally valid attributesfor grouping patterns, such as object color or orientation.
As shown in [29, 30], novel class discovery(NCD) can be well-posed and tackled in a data-driven manner by injecting prior knowledge on howwe wish to group semantic classes using some degree of supervision.
Existing methods [29, 30,26, 25, 72, 7, 73, 74, 19] all assume curated datasets, where objects of interest are pre-cropped andsemantic classes are fairly balanced.
In contrast, we tackle joint novel class discovery and localization(NCDL) , a task of learning to discover and detect objects from raw, unlabeled, and uncurated data.
This is a significantly more challenging problem, as images always contain a mixture of labeled andunlabeled object classes, and we need to localize as well as categorize them.
To study this problem, we re-purpose the COCO [44] and LVIS [24] datasets.
We use 50% of COCOimages together with COCO class vocabulary and labels for supervised model training.
We use theremaining 50% of images without any labels during the training and evaluate how well our networklearns to detect novel classes using the long-tailed LVIS label set.
This setting allows us to mimic real-world scenarios where no supervision is given for the target domain, while a rich set of labelsallows us to rigorously study this problem.
We train a two-stage object detector [53, 27] in an end-to-end manner such that (i) we can correctlydetect ( i.e. localize and categorize) labeled objects, and, in parallel, (ii) detect unlabeled objectsby learning feature representations suitable for categorization of objects that would otherwise beclassified as the background class.
We start with supervised training on the labeled, source domain,followed by the self-supervised training on the target (unlabeled) domain.
We transfer knowledgefrom the source to the target domain by retaining weights for class-agnostic modules, such as RPNand bounding box regression, and instance segmentation heads.
For classification, we reuse theprimary classification head and add a new secondary head, training them jointly with an objectiveto categorize every region proposal consistently under multiple transformations and a constraintthat the marginal distribution of class assignments follows a long-tail distribution.
By encouragingthe network to learn features for a variety of objects via a non-uniform classification prior, we alsoeffectively reduce the bias towards simply classifying regions as one of the labeled or backgroundclasses.
Our end-to-end trainable method significantly outperforms prior art [68] that learns object representations using contrastive learning followed by k-means clustering, while providing categorizationfor any object with a single network forward pass.
More importantly, with this work, we lay downthe foundations for an exciting new line of research that goes beyond well-established supervisedlearning for object detection.
In summary, this paper makes the following contributions : (i)
We study the problem of novel classdiscovery in the wild using standard object detection datasets that do not pre-localize regions ofinterest manually or assume objects are roughly centered, i.e. , photographer bias.
(ii) We proposea simple yet effective and scalable method that can be trained end-to-end on the unlabeled domain.
Our network is bootstrapped using a labeled dataset to learn features suitable for categorizing futureinstances.
We train feature representations for arbitrary objects using the self-supervised objective.
Finally (iii), we reveal that our method can discover and detect novel class instances and outperformprior work, demonstrating the generalization to datasets beyond COCO.
Related Work
Object discovery in image and video collections.
Discovery of semantic objects that appear inunlabeled video collections is a long-standing research problem [60].
Early methods employ multiplebottom-up segmentations of images [55, 59] to discover commonly occurring patterns using topicmodeling, e.g. , Dirichlet allocation [6].
Lee et al.
propose to rely on saliency [41] or to incorporatesemantic knowledge about certain semantic classes, for which supervision is available [42].
Thesemethods are evaluated on smaller datasets [23, 58, 17] that contain one or a handful of well-delineatedobjects.
Rubinstein et al.
[54] demonstrated that discovery can be scaled to internet photo collections.
Beyond images, Kwak et al.
[40] tackle the discovery of objects in YouTube videos that usuallycontain a single object exhibiting dominant motion.
They build on a two-stage approach: (i) identifythe most salient region proposals in terms of appearance and motion, and (ii) co-segment consistentlyappearing objects across video clips.
Video object discovery videos, recorded from a moving platform,were tackled in [49, 50] by first mining video-object tracks by associating region proposal network(RPN) [53] based proposals across time, followed by clustering of video proposals based on pretrained CNN features.
Recent work on open-set panoptic segmentation [32] similarly employsclustering based on RPN features to discover commonly-occurring objects, used as pseudo-labels forlearning to segment unlabeled objects.
The method by [68] follows the general approach of clustering
RPN proposals, however, instead of relying on pre-trained CNN features, they learn representationssuitable for clustering using self-supervised objective, trained via contrastive learning.
While most of the foregoing works focus on saliency criteria to significantly narrow down the numberof region proposals, [62, 63] resort to combinatorial optimization that can cope with a large set ofobject proposals to discover objects that frequently appear in image datasets.
It has been shown [64]that this approach can be scaled to large image collections, e.g. , OpenImages [39].
Novel class discovery.
In Novel class discovery (NCD) works, given is labeled data that can be usedto train a feature extractor and learn a clustering criterion to guide a notion of similarity and unlabeleddata is assumed to contain novel class instances only.
Hsu et al.
[29, 30] propose a frameworkthat injects prior knowledge on how we wish to group semantic classes from supervised learningon labeled images to the unlabeled domain via learnable similarity function.
The work of Han etal.
[26] builds on deep clustering [71] and suggests an approach for automatically discovering thenumber of novel classes.
In follow-up work, Han et al.
[25] build on pairwise similarity of imagefeatures [12] and propose leveraging self-supervised pretraining along with freezing the backboneto reduce bias towards the known classes.
Its follow-up works [72, 34, 74, 73] further boost theperformance by introducing additional self-supervised and regularization components.
UNO [19]proposes a simple method based on SwAV [11] that bootstraps signal for novel class training usingpseudo-labels generated under equipartitioning constraints.
ORCA [7] extends the NCD problemand operates under the assumption that unlabeled images can contain instances of known classes aswell.
Motivated by [61], [12], [65], authors leverage self-supervised initialization of backbone,supervised loss along with unsupervised loss based on pairwise pseudo-labels, and novel dynamicsoftmax margin [65] to alleviate bias of the known classes.
In contrast to our work, the foregoingmethods investigate the problem in the image classification setting, assuming objects of interest arelocalized (pre-cropped), and operate under absence of outliers ( e.g. , images containing no objects).
Related and complementary research directions.
Methods for generic [45] or large vocabularyobject detection [24] focus on detecting/segmenting a large set of a-priori known semantic classes,often by utilizing multiple datasets and (possibly weak) supervision [75, 52, 31].
A special case iszero-shot learning [70] and zero-shot object detection [3, 48, 51], where (mostly weak) supervisionfor unseen classes comes in the form of attributes, class names, latent features, or other forms ofauxiliary data, commonly used in a conjunction with language models embeddings to learn a jointembedding space. In few-shot learning [36, 66, 18, 8] given is a limited supervision, where algorithmsare required to learn from one or a few data samples per each class. In [18, 8] authors employ asecondary classification head that is trained using few-shot supervision and distillation [18] or marginloss [8].
In contrast to zero-shot, few-shot, and large-vocabulary detection, we assume no supervisionand prior knowledge for object classes that appear in the tail.
Methods for open-set recognition [56, 57, 33, 5] and detection [48, 16] focus on calibration ofper-class uncertainties to minimize the confusion between known and possibly unknown classes.
Open-world recognition methods, as defined by [4, 46], must explicitly recognize unknown objectinstances that were not observed as labeled samples during the model training, and must continuallyupdate object detectors to recognize these unknown instances.
Recent method for open-worlddetection [35] focuses on the continual learning aspect of the task: for the evaluation, labels for“novel” classes are fully provided (those are the held-out classes).
Our work is orthogonal, andtackles the “missing bit” of the open-world detection pipeline: discovery and semantic grouping ofnovel -class objects that can appear in our training data as well, however, are unlabeled.
Clusters ofdiscovered novel classes could be verified by human annotators and used to update detection modelsin a continual fashion, as proposed by [4].
Novel Class Discovery and Localization
We first detail novel class discovery and localization (NCDL) , the task of learning to discover anddetect objects from raw, unlabeled, and uncurated data.
Given is a source ( labeled ) object detectionimage dataset D k for ( known ) object classes C k ( | C k | = K ), that provides both class and box levelobjects supervision: D k
= { ( I , { ( b , c ) } ) } , where I ∈ R h × w × 3 , b ∈ R 4 , and c ∈ C k .
In addition,given is one or more target ( unlabeled ) datasets D n = { I } that may contain instances of bothpreviously seen and unseen ( novel ) object classes C k ∪ C n ( | C n | = N, N > K ) that we need tolearn to detect in the absence of any labels.
For an arbitrary image I from the source or target domain, NCDL methods should detect all object instances present in the image: { ( b ′ , c ′ ) } , i.e. localize themalong with predicting their semantic class ( e.g. , a car needs to be detected as the car class).
As the main contribution of this paper, we propose an end-to-end trainable detection and discoverynetwork that jointly learns discriminative representations for all region proposals, their localization,and categorization alleviating the need for explicit clustering after the network training.
We build ona two-stage Faster or Mask R-CNN [53, 27].
Such detectors are naturally suitable for NCDL as theirregion proposal network (RPN) is trained in a class-agnostic manner to propose regions of interestthat likely contain objects.
Given object-centric feature representations for each localized candidateregion, that we learn via supervised pre-training, one natural approach for discovering (categorizing)novel classes would be to cluster such regions in feature space using, e.g. , k-means, c.f. , [49, 32, 68].
However, leveraging feature representation learned purely via supervised R-CNN training is likely tonot be discriminative enough with respect to objects of unlabeled classes.
Moreover, such approachis prone to having a strong bias toward classifying novel instances as one of the labeled classes.
Tothis end, we introduce our end-to-end RNCDL framework that employs pseudo-labeling obtained viaconstrained clustering. Via a newly attached classification head, it learns to discriminate objects ofall classes, including the unlabeled ones, building on learned feture representations.
Supervised training on the source domain.
We begin with a self-supervised backbone pretraining [28] on a large-scale image dataset, such as ImageNet [15], and use it to initialize thebackbone for an R-CNN network. We then train the object detector on the source (labeled) domainusing supervision for objects of known C k classes, such as 80 COCO classes (Fig.
2A, top ).
Weuse the labeled data to train the shared feature extractor (backbone) together with a class-agnosticregion proposal network (RPN), class-agnostic box regression head, as well as a classification head for categorization of C k classes.
Specifically, for each image I , RPN is trained to predict a setof class-agnostic proposals { ( b , o ) } , where b ∈ R 4 are predicted coordinates and o ∈ R is theobjectness score. We then pass the proposals’ feature representations, obtained from the backbone viathe RoIAlign [27] mechanism, through an MLP (box head) and pass the resulting features ˆ f ∈ R ˆ F to RoI heads for the final predictions.
We train a class-agnostic bounding box head to refine box locationfor each object ˆ b = h bb ( b , ˆ f ) , and a classification box head to categorize each object into one ofthe K classes or a background: ˆ c = h k (ˆ f ) = W k ˆ f , where ˆ c ∈ R K +1 and W k ∈ R ( K +1) × ˆ F .
Fortraining we use a standard fully-supervised R-CNN loss: L sup =
L RPN + L box + L cls , consistingof the sum of RPN, localization, and classification losses respectively [53, 27].
Discovery on the target domain.
During the following self-supervised (discovery) training phaseon the target domain, we attach a secondary (novel) classification head h n on top of RoI box features 172 ˆ f (Fig. 2B). The secondary classification head consists of an MLP and a linear classification layerthat we detail below. We also modify the primary (known) classification head, by removing thebackground class from its weight matrix such that ˆ W k ∈ R K × ˆ F .
As the background class isremoved at this phase, we aim to learn to cluster proposals. To obtain classification predictionsat this phase, we pass RoI features through both h k and h n classification heads, concatenatingtheir logits: ˆ p = softmax ([ l k , l n ]) = softmax ([ h k (ˆ f ) , h n (ˆ f )]) ∈ R K + N . During training, wefreeze all layers except the classification heads.
We train the classification heads jointly using aself-supervised classification loss and supervised classification loss: L disc =
L ss + α · L cls , whereα is the supervised loss scale coefficient. We show in ablations that keeping the downscaled ( α < 1 )supervised classification loss in the loop effectively alleviates forgetting the weights for the knownclasses, while large α may create an unwanted bias and harm learning of novel classes.
As we removethe background class, different from the supervised phase, we calculate the supervised loss only forannotation-matched region proposals. For the self-supervised classification loss we use cross-entropyloss on top of pseudo-labels generated online for the current batch of proposals that we detail in asubsection below.
The employed loss encourages the network to categorize every region proposalconsistently under multiple transformations and a constraint that the marginal distribution of predictedassignments for the last batches follows a prior long-tail distribution (Fig.
2A, bottom ).
With such ajoint training objective, our network is encouraged to learn to distinguish a large variety of classespresent in the regions sampled by RPN, while retaining the capability to classify known objects. Secondary classification head architecture.
As the box features ˆ f are kept frozen during thediscovery phase, we use several non-linear projection layers in the secondary classification head: 193 ˆ f n = g n (ˆ f ) , where g n is a multi-layer perceptron with ReLU activations. Such learned transformationis designed to better disentangle feature representation of novel classes without affecting the RoIbox features that the known head is based on. On top of the projected features ˆ f n we employ acosine classification layer, shown to be effective in the context of few-shot learning [20, 13, 67]:l n = W n || W n || · ˆ f n ||
ˆ f n || , where l n ∈ R N are the predicted novel-class logits. We determine the number ofnovel classes ( i.e. the number of novel classes, N , in the secondary head) empirically in (Sec. 4.2).
Self-supervision via constrained clustering. We perform self-labeling via constrained clustering, where we assign a soft pseudo-label to each RPN proposal. We generate pseudo-labels q forall input images from D N ∪ D K that span all classes, both known and novel: q ∈ R ( N + K ) × B ,where B is the batch size. We use q to compute the cross-entropy loss L ss and train the classification heads.
Standard clustering methods ( e.g. k-means [47]) were shown to produce unstableresults when employed for pseudo-labeling [9, 10].
Asano et al [2] instead propose to leveragean equipartitioning regularizer and [1] further extended the method to arbitrary prior distributions. Following [1], we aim to minimize a clustering energy function ( i.e. cross-entropy loss): E (ˆ p , q ) = 1 B (cid:80) Bi =1 (cid:80) N + K y q ( y
|
I i ) log ˆ p ( y | I i ) , where q are class labels ( i.e. pseudo-labels in ourframework), and ˆ p = softmax ([ l k , l n ]) ∈ R ( N + K ) × B are predicted class-probabilities.
Note thatboth labels q and probabilities ˆ p are subject to optimization. Similar to Asano et al [2], we employ an alternating algorithm, where a constrained clustering method to generate q is followed byoptimizing ˆ p , i.e. , during each training iteration, we first generate class pseudo-labels q , and thenuse them to train the classification heads.
To generate class pseudo-labels, we first choose a targetclusters’ marginal probability distribution ( i.e. prior distribution). In our experiments, as prior, we uselognormal distribution.
We then generate such soft pseudo-labels q that (i) closely match the chosenmarginals and (ii) minimize E (ˆ p , q ) for the given ˆ p . In [2, 1], it is shown that this can be posed as a constrained optimization problem and solved as an optimal transport problem using the fast online
Sinkhorn-Knopp algorithm
Having obtained the pseudo-labels q , we fix them, calculate theloss L ss =
E (ˆ p , q ) , and apply backpropagation to the classification heads’ parameters.
Memory module.
To generate good-quality pseudo-labels, we need a diverse set of samples inmini-batches in order to capture a wide variety of classes. With only a small set of features permini-batch, we would likely produce noisy cluster assignments.
As we operate in scenarios withmore than a thousand classes to discover, we introduce a memory module ( c.f. , [28, 11]) to storeproposals’ RoI box features from the last batches and ensure diverse set of samples when computingclustering assignments.
As in [28, 11] we design the memory module as a queue, where at theend of each iteration we replace the old set of stored features with the features from the currentbatch. During pseudo-labeling generation we pass the extended set of features to the Sinkhornalgorithm. Specifically, after extracting current-batch proposals’ features, we concatenate them withthe features stored in the memory module M , compute their logits, and as an input to the Sinkhornalgorithm use: ˆ l = [ˆ l k , ˆ l n ] = [ h k ([ˆ f , M ]) , h n ([ˆ f , M ])] , where M ∈ R M sz × ˆ F and M sz is the numberof features stored in the memory.
Such modified procedure generates B + M sz pseudo-labels, forboth current-batch proposals and memory features. We discard pseudo-labels generated for memoryfeatures and use only those B generated for the current-batch proposals, i.e. , we use the memorymodule only to improve the quality of current-batch pseudo-labels.
Swapping multi-view cluster assignments. We employ supervisory signal from multiple augmentations [11] to enforce the network to output consistent classification predictions ˆ p . Specifically,for each image from D N ∪ D K , we first generate RPN proposals and refine their coordinates viaclass-agnostic classification head.
We then regress the proposals’ coordinates, and generate it’stwo augmented views (we discuss augmentations in the supplementary), and extract features foreach. Then, we generate pseudo-labels for each set of features individually, obtaining two sets ofpseudo-labels q i , i ∈ { 1 , 2 } . Now, instead of computing the losses for each view as E (ˆ p i , q i ) , weswap the resulting pseudo-labels q i between views, and calculate losses as E (ˆ p i , q j ) , where i ̸ = j ,i.e. we use pseudo-labels obtained for one view, as labels to the other.
We calculate the total loss asthe average of the losses for both views: ˆ L ss = ( E (ˆ p 1 , q 2 ) + E (ˆ p 2 , q 1 )) / 2 . For each view, we usea separate feature memory module.
Inference and evaluation. During inference (Fig. 2B), for each image, we extract a set of RPNproposals { ( b , o ) } and pass them through RoI heads for localization refinement and class predictions.
For classification, we use both heads h k , h n and calculate probabilities as ˆ p = softmax ([ l k , l n ]) .
For quantitative evaluation, as our framework provides arbitrary class predictions ( i.e. not directlymapped to novel classes in the target domain), we have to obtain a mapping of predicted classes withground truth labels. We follow prior art [26, 25, 19, 68] and propose a similar evaluation methodology. First, we generate classification predictions for each GT annotation in the validation dataset.
Then, weuse the Hungarian algorithm [38] to obtain an one-to-one mapping between GT labels and validationinstances based on these class predictions.
Given this mapping, we ignore instances of predictedclasses that were not mapped to GT classes. We proceed to the standard R-CNN inference pass, wherewe apply the foregoing mapping right after generating class predictions, and follow the standard R-CNN post-processing and evaluation [44].  Experimental Evaluation
In this section, we discuss our evaluation test-bed, including datasets, evaluation settings, and metrics(Sec. 4.1).
In Sec.
4.2 we justify our main design decisions by studying NCDL performance in awell-controlled setting that closely follows a real-world scenario (training on a source labeled andtarget unlabeled dataset).
We then compare our method’s performance to several baselines (Sec. 4.3)and, finally, demonstrate that our method is applicable beyond our evaluation test-bed (Sec. 4.4).
4.1 Evaluation setting
COCO half + LVIS.
We re-purpose COCO 2017 [44] and LVIS v1 [24] datasets for running ablationsand comparisons with baselines and prior art.
We use annotations for 80 COCO classes during thesupervised training phase, aiming at further classifying additional 1000+ LVIS classes.
COCOdataset contains 123 K images with modal bounding box and segmentation mask annotations for 80classes.
The LVIS dataset contains a subset of 120 K COCO images with annotations for 1203 classes We check the results of our method as function of strength of the supervised loss. [Table]
[Table] that include all classes from COCO. For our experiments, we follow LVIS training and validationsplits, resulting in 100 K training images and 20 K validation images.
To model the NCDL setup, wefurther split 100 K training images in half and use only 50 K images with 80 -class annotations from
COCO during the supervised training phase.
We refer to this dataset as COCO half .
We treat the restof 50 K images as additional unlabeled data used during the discovery phase.
LVIS + Visual Genome.
We perform a large-scale generalization experiment by using annotationsfor 1203 LVIS classes during the supervised learning ( i.e. , treating LVIS classes as labeled ), aimingto learn to discover extra 2700 + classes from the Visual Genome (VG)
v1.4 dataset [37] ( i.e. , treating
VG classes as unlabeled ).
We provide dataset details and splits (that ensure LVIS and VG classvocabularies do not overlap) in the supplementary.
We note that annotations provided in VG are notexhaustive per class and many of its classes are abstract or semantically overlapping both within VGand when compared to LVIS.
We thus mainly focus on qualitative results.
Implementation details.
We base our model on Mask R-CNN [27] and FPN [43] implementationsfrom Detectron2 [69].
We give all the implementation details in the supplementary material.
Evaluation metrics.
For quantitative evaluation, we follow [26, 25, 19, 68] to match predictedcluster assignments with annotated semantic classes.
We follow common practice and report meanaverage precision (mAP@[.5:.95]) [44].
For qualitative results, we include instances of classes thatwere matched to annotated classes only.
We detail our evaluation procedure in the supplementary.
Baselines.
We compare our end-to-end trainable detector with state-of-the-art Weng et al.
[68]work, NCD methods ORCA [7], UNO[19], and k-means [47] baseline that operate on cropped objectinstances.
For NCD methods we use labeled instance annotations from COCO half dataset as labeledimages pool and cropped RPN proposals as unlabeled images pool.
For proposals extraction wetrain R-CNN on COCO half under the same conditions as our base network, to ensure comparableevaluation conditions.
Having trained the NCD classifiers, we use the same R-CNN to generate objectproposals for the validation images, apply the classifiers on top of the proposals, and continue with thestandard R-CNN post-processing steps.
We provide more details on baselines in the supplementary.
4.2 Model ablations
We ablate single and multi-phase training, the impact of keeping supervised loss in the loop, thenumber of novel categories, pseudo-labeling prior, model architecture decisions, and backbonepretraining.
In supplementary we provide further results on experiments with hyperparameters forproposals extractor and how class-agnostic heads affect the performance of the supervised baseline.
Single-phase training and unfreezing R-CNN components during discovery.
As shown in Table3 we observe that training the network with both supervised and discovery losses from the beginning,summed up without any scaling, leads to divergence.
We hypothesize that this is due to noisygradients from our self-supervised signal, with R-CNN being very sensitive to the choice of lossesand hyperparameters [53].
To further support this claim, we experiment with a standard supervisedphase followed by a discovery phase without freezing any of the components.
We observe that such asetup also leads to divergence.
Unfreezing only the RoI heads during the discovery phase, however,does not lead to divergence, yet reduces the score by 1 . 32 mAP.
We thus keep all the componentsfrozen during the discovery phase, apart from known and novel classification heads.
The strength of the supervised loss.
We ablate the strength of the supervised classification lossthat is used along with the discovery loss during the discovery phase.
We vary the strength from0 (no supervised loss) to 1 (no downscaling) and provide the results in Table 1.
We observe thatexcluding supervised loss completely degrades the performance of the network for the known classesand benefits the performance of the novel classes.
This indicates that there is a tradeoff betweenperformance on known and novel classes, - keeping the full supervised loss introduces too much biastowards the former.
We consequently set the strength of the supervised loss to 0 .
5 in all experiments.
[Table]
[Table]
Sensitivity to the number of novel classes.
In Table 2 we experiment with varying the number ofnovel classes we use for the novel classification head.
We observe that the best number of classesfor the COCO + LVIS setup is 3000 , while other choices result in lower scores.
We conjecture thatsuch scores are sensitive to the semantic classes defined by the target dataset - with 3000 classes themodel learns the granularity and grouping that matches closest to the LVIS classes, but that does notnecessarily mean that a finer or coarser granularity categorization is worse.
In supplementary we alsoexperiment with attaching and training multiple heads jointly during the discovery phase and observethat scores slightly degrade in such setup but do not observe a significant drop in results.
Clustering prior.
In Table 3 we experiment with using uniform prior distribution for class-marginalsduring pseudo-labels generation, as proposed in [2, 10].
We observe degradation of the score by1 .
17 mAP, which confirms our hypothesis that a non-uniform lognormal prior distribution closerresembles a real-world long-tail class distribution.
As an additional ablation, we experiment withusing k-means for online pseudo-labels generation instead of constrained clustering and observe asignificant degradation of the score by 4.86 mAP, especially notable for the novel classes.
Embedding projector and swapped assignments.
In Table 3 we demonstrate the effect of addingan MLP feature projector on top of RoI box features for novel heads classification.
We observe thatthe performance of the network improves by 1 . 62 mAP.
In the same table, we show that swappingpseudo-labels between proposals under multiple augmentations helps boost the score by 0 .
18 mAP.
Feature memory module.
We demonstrate the importance of memory module in Table 3.
In allour experiments we used the memory size of 100 batches, unless stated otherwise.
We observe thatremoving the memory module significantly drops the performance by 4 . 09 mAP, especially for thenovel classes.
This highlights that for an efficient pseudo-labels generation it is crucial to use a set offeatures representative of the full feature space that we lack in the absence of memory.
Self-supervised pre-training matters.
We compare the results of our framework under differentbackbone initialization employed for the supervised pre-training stage.
For the randomly initializedbackbone we set all layers as trainable, while during the MoCo initialization the first two convolutionalblocks are kept frozen ( c.f. , [28]).
In Table 3 we observe that MoCo-initialized backbone improvesthe score by 2 . 15 mAP over randomly initialized backbone mode.
Proposals extractor hyperparameters.
We observe that when extracting the RPN proposals forself-supervised learning during the discovery phase the absence of NMS yields the best results and theoptimal number of proposals per image is 50.
We thus use such parameters for all the experiments.
4.3 Comparison to prior work
In Table 4 we compare our method with k-means baselines and three recent state-of-the-art methodsthat we adapted to our scenario.
We present the results for both known (COCO) and unknown (the restof LVIS) classes.
Our method significantly outperforms the previous best NCD method UNO [19],by reaching 4 .
74 higher overall mAP.
We outperform UNO in both the known and especially novelclasses, where we reach 4 .
81 higher mAP.
We also significantly outperform approach by Weng etal.
[68].Our method is the only method to reach over 1 mAP in novel classes, where we reach 5 . 42mAP.
We believe the main advantages of our method to be the memory module, critical for effectiveself-supervision, and a backbone trained in an end-to-end manner with the aid of class-agnosticlosses, further frozen during the discovery phase.
Further, our backbone benefits from high-levelsemantic features via FPN that is critical for the classification of small objects [43].
We provide more comparison details in the supplementary.
We note that as our method heavily relies on thegenerated region proposals, we are limited by the performance of region proposal network.
It alsodoes not update the backbone feature extractor, which could further boost the results, including theperformance of localization and segmentation components.
4.4 Cross-dataset Generalization and Qualitative Results
We analyze the generalization properties of our framework by training on LVIS and evaluating on VG dataset. We present quantitative results in Table 5. a fully supervised method.
[Table]
In Figure 3 we provide qualitative results withnovel classes discovered.
In Figure 4 we alsovisualize the common discovered categories.
Upon manual examination of the clusters fromthe LVIS + VG experiment, we observe thatmany of the novel classes are synonyms, hyponyms, or hypernyms of known classes thatappear in the LVIS dataset.
Conclusions
In this paper, we introduce an end-to-end RNCDL network for the task of novel class discovery,detection, and localization.
Our model is a two-stage object detection network that is able to classifyboth instances of the labeled, known classes and those of unlabeled, novel classes.
At the core of ourframework is a self-supervision guided by features of region proposals and a constraint for the novelclass assignments to follow a long-tail distribution.
In our experiments we demonstrate a significantimprovement over the previous state-of-the-art.
Furthermore, we demonstrate the ability to detectsemantic classes without any supervision at a large scale on the Visual Genome dataset.
Broader Impact
The success of deep learning, to a large extent, can be attributed to high-quality labeled datasets.
These datasets, for the most part, contain only a fixed number of categories (classes).
Deep learningmodels trained on such datasets can only make predictions for objects that belong to one of thetraining set categories.
However, the world consists of an extremely large number of categories,and enlarging the training set with more labeled data for all the categories is an infeasible problem.
Clearly, there is a need of shit towards solutions that are not limited by labeled data supervision.
Novel class category discovery has been mostly tackled in image classification setup, but there havebeen no works that give suitable solutions to the more challenging task of object detection.
In thiswork, we improve compared to previous methods by an order of magnitude.
While the problem is stillfar from solved, we hope that our method makes the first concrete step toward an effective solution,and would inspire others to tackle this challenging problem.
We expect that this method will lead tomore object detection works that aim to discover instances of novel classes.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes]
(c) Did you report error bars (e g., with respect to the random seed after running experiments multiple times)?
[No](d) Did you include the total amount of compute and the type of resources used (e g., typeof GPUs, internal cluster, or cloud provider)?
[Yes]4.
If you are using existing assets (e g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[Yes]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]