Learning to Discover and Detect Objects
Anonymous Author(s)
AffiliationAddress email
Abstract
We tackle the problem of novel class discovery, detection, and localization (NCDL).
In this setting, we assume a source dataset with labels for objects of commonlyobserved classes.
Instances of other classes need to be discovered, classified, andlocalized automatically based on visual similarity, without human supervision.
Tothis end, we propose a two-stage object detection network Region-based NCDL( RNCDL ), that uses a region proposal network to localize potential objects andclassify them.
We train our network to classify each proposal, either as one of theknown classes, seen in the source dataset, or one of the extended set of novel classeswith a constraint that the distribution of class assignments should follow naturallong-tail distributions common in the real open-world.
By training our detectionnetwork with this objective in an end-to-end manner, it learns to classify all regionproposals for a large variety of classes, including those that are not part of thelabeled object class vocabulary.
Our experiments conducted using COCO and LVISdatasets reveal that our method is significantly more effective compared to multistage pipelines that rely on traditional clustering algorithms or self-supervisedcontrastive learning methods and operate on pre-extracted crops.
Beyond that, wedemonstrate the generality of our approach by applying our method to a large-scale
Visual Genome dataset, where our network successfully learns to detect varioussemantic classes without explicit supervision.
Introduction
We tackle novel class discovery and localization in unlabeled datasets, a long-standing problemin computer vision [51, 55, 37, 38, 50].
As we are approaching the limits of well-understood andsuccessful supervised training of object detectors [20, 19, 49, 25], this is becoming an increasinglymore important problem, with the task of labeling novel and rare classes that appear in the long tailof object class distribution becoming prohibitively expensive [22].
Object class discovery is a difficult problem, as there may be many possible equally valid attributesfor grouping patterns, such as object color or orientation.
As shown in [27, 28], novel class discovery(NCD) can be well-posed and tackled in a data-driven manner by injecting prior knowledge on howwe wish to group semantic classes using some degree of supervision.
Existing methods [27, 28,24, 23, 67, 7, 68, 69, 17] all assume curated datasets, where objects of interest are pre-cropped andsemantic classes are fairly balanced.
In contrast, we tackle joint novel class discovery and localization(NCDL) , a task of learning to discover and detect objects from raw, unlabeled, and uncurated data.
This is a significantly more challenging problem, as images always contain a mixture of labeled andunlabeled object classes, and we need to localize as well as categorize them.
To study this problem, we re-purpose the COCO [40] and LVIS [22] datasets.
We use 50% of COCOimages together with COCO class vocabulary and labels for supervised model training.
We use theremaining 50% of images without any labels during the training and evaluate how well our networklearns to detect novel classes using the long-tailed LVIS label set.
This setting allows us to mimic real-world scenarios where no supervision is given for the target domain, while a rich set of labelsallows us to rigorously study this problem.
We train a two-stage object detector [49, 25] in an end-to-end manner such that (i) we can correctlylocalize and categorize, i.e. , detect labeled objects, and, in parallel, (ii) learn feature representations,suitable for categorization of objects of novel categories that would otherwise be classified as thebackground class.
We start with supervised training on the labeled, source domain, followed by theself-supervised training on the target (unlabeled) domain.
We transfer knowledge from the source tothe target domain by retaining weights for class-agnostic modules, such as RPN and bounding boxregression, and instance segmentation heads.
For classification, we reuse the primary classificationhead and add a new secondary head, training them jointly with an objective to categorize every regionproposal consistently under multiple transformations and a constraint that the marginal distributionof class assignments follows a long-tail distribution.
By encouraging the network to learn features fora variety of objects via a non-uniform classification prior, we also effectively reduce the bias towardssimply classifying regions as one of the labeled or background classes.
Our end-to-end trainable method significantly outperforms prior art [63] that learns object representations using contrastive learning followed by k-means clustering, while providing categorizationfor any object with a single network forward pass.
More importantly, with this work, we lay downthe foundations for an exciting new line of research that goes beyond well-established supervisedlearning for object detection.
In summary, this paper makes the following contributions : (i)
We study the problem of novel classdiscovery in the wild using standard object detection datasets that do not pre-localize regions ofinterest manually or assume objects are roughly centered, i.e. , photographer bias.
(ii) We proposea simple yet effective and scalable method that can be trained end-to-end on the unlabeled domain.
Our network is bootstrapped using a labeled dataset to learn features suitable for categorizing futureinstances.
We train feature representations for arbitrary objects using the self-supervised objective.
Finally (iii), we reveal that our method can discover and detect novel class instances and outperformprior work, demonstrating the generalization to datasets beyond COCO.
Related Work
Object discovery in image and video collections.
Discovery of semantic objects that appear inunlabeled video collections is a long-standing research problem [56].
Early methods employ multiplebottom-up segmentations of images [51, 55] to discover commonly occurring patterns using topicmodeling, e.g. , Dirichlet allocation [6].
Lee et al.
propose to rely on saliency [37] or to incorporatesemantic knowledge about certain semantic classes, for which supervision is available [38].
Thesemethods are evaluated on smaller datasets [21, 54, 16] that contain one or a handful of well-delineatedobjects.
Rubinstein et al.
[50] demonstrated that discovery can be scaled to internet photo collections.
Beyond images, Kwak et al.
[36] tackle the discovery of objects in YouTube videos that usuallycontain a single object exhibiting dominant motion.
They build on a two-stage approach: (i) identifythe most salient region proposals in terms of appearance and motion, and (ii) co-segment consistentlyappearing objects across video clips.
Video object discovery videos, recorded from a moving platform,were tackled in [45, 46] by first mining video-object tracks by associating region proposal network(RPN) [49] based proposals across time, followed by clustering of video proposals based on pretrained CNN features.
Recent work on open-set panoptic segmentation [30] similarly employsclustering based on RPN features to discover commonly-occurring objects, used as pseudo-labels forlearning to segment unlabeled objects.
The method by [63] follows the general approach of clustering
RPN proposals, however, instead of relying on pre-trained CNN features, they learn representationssuitable for clustering using self-supervised objective, trained via contrastive learning.
While most of the foregoing works focus on saliency criteria to significantly narrow down the numberof region proposals, [58, 59] resort to combinatorial optimization that can cope with a large set ofobject proposals to discover objects that frequently appear in image datasets.
It has been shown [60]that this approach can be scaled to large image collections, e.g. , OpenImages [35].
Novel class discovery.
Works from a related Novel class discovery (NCD) field operate under anassumption that given is a labeled data that can be leveraged to train a feature extractor along withlearning a clustering criterion to guide a notion of similarity and unlabeled data that is assumedto contain novel class instances only.
Hsu et al.
[27, 28] propose a framework that injects priorknowledge on how we wish to group semantic classes from supervised learning on labeled images bytransferring knowledge to the unlabeled domain via learnable similarity function.
The work of Hanet al.
[24] builds on deep clustering [66] and suggests an approach for automatically discovering thenumber of novel classes.
In follow-up work, Han et al.
[23] suggest an alternative training objectivebased on pairwise similarity of image features [11] using pseudo-labels for fine-tuning the networkand propose leveraging self-supervised pretraining and freezing the backbone to reduce bias towardsthe known classes.
Its follow-up works [67, 32, 69, 68] further boost the performance by introducingadditional self-supervised and regularization components.
UNO [17] proposes a simple method basedon SwAV [10] that bootstraps signal for novel class training using pseudo-labels generated underequipartitioning constraints.
ORCA [7] extends the NCD problem and operates under the assumptionthat unlabeled images can contain instances of known classes as well.
Motivated by [57], [11], [61],authors leverage self-supervised initialization of backbone, supervised loss along with unsupervisedloss based on pairwise pseudo-labels, and novel dynamic softmax margin [61] to alleviate bias ofthe known classes.
In contrast to our work, the foregoing methods investigate the problem in theimage classification setting, assuming objects of interest are localized (pre-cropped), and operateunder absence of outliers ( e.g. , images containing no objects).
Related and complementary research directions.
Methods for generic [41] or large vocabularyobject detection [22] focus on detecting/segmenting a large set of a-priori known semantic classes,often by utilizing multiple datasets and (possibly weak) supervision [70, 48, 29].
A special case iszero-shot learning [65] and object detection [3, 44, 47], where (very weak) supervision for unseenclasses comes in the form of attributes, or class names, commonly used in a conjunction with languagemodels embeddings to learn a joint embedding space, however, all the target classes are known andpre-determined.
In contrast to zero-shot and large-vocabulary detection, we assume no supervisionand prior knowledge for object classes that appear in the tail.
Methods for open-set recognition [52, 53, 31, 5] and detection [44, 15] focus on calibration ofper-class uncertainties to minimize the confusion between known and possibly unknown classes.
Open-world recognition methods, as defined by [4, 42] must explicitly recognize unknown objectinstances that were not observed as labeled samples during the model training, and must continuallyupdate object detectors to recognize these unknown instances.
Recent method for open-worlddetection [33] focuses on the continual learning aspect of the task: for the evaluation, labels for“novel” classes are fully provided (those are the held-out classes).
Our work is orthogonal, andtackles the “missing bit” of the open-world detection pipeline: discovery and semantic grouping ofnovel -class objects that can appear in our training data as well, however, are unlabeled.
Clusters ofdiscovered novel classes could be verified by human annotators and used to update detection modelsin a continual fashion, as proposed by [4].
Novel Class Discovery and Localization
In this section, we detail novel class discovery and localization (NCDL) , the task of learning todiscover and detect objects from raw, unlabeled, and uncurated data.
During the model training, weassume we are given a source ( labeled ) dataset that provides bounding box or segmentation masklevel supervision for K object classes.
This data can be used to learn features useful for categorizationand other aspects of the task, such as bounding box regression and mask segmentation, which canbe learned in a class-agnostic manner.
In addition, we are given one (or more) unlabeled datasetsthat may contain instances of both previously seen and N, N > K visual object classes that weneed to learn to detect in the absence of any labels.
Given an arbitrary image from the source ortarget domain, methods should detect and localize all object instances present in the image, alongwith predicting their respective semantic class ( i.e. , a car needs to be detected as the car class).
Asthe main contribution of this paper, we propose an end-to-end trainable detection and discoverynetwork that jointly learns discriminative representations for all region proposals, their localization,and categorization alleviating the need for explicit clustering after the network training. 3.1 Region-based Novel Category Discovery and Localization (RNCDL)
We start with a two-stage Faster or Mask RCNN [49, 25].
Such detectors are naturally suitable for NCDL as the region proposal network (RPN) is trained in a class-agnostic manner to propose avariety of regions (anchor boxes) of interest that likely contain objects.
Given object-centric featurerepresentations for each candidate region that can be learned via supervised training, one naturalapproach for discovering novel classes would be to cluster such regions in feature space using, e.g. ,k-means, c.f. , [45, 30, 63].
However, in the following, we argue that feature representation learnedvia supervised RCNN training is not trained to be discriminative with respect to objects of unlabeledclasses, as every anchor box that does not have sufficient overlap with labeled boxes is classifiedas background class during training. Via this process, feature representations for all regions thatdo not contain objects of labeled classes are effectively collapsed and not discriminative for robustcategorization of novel classes.
Furthermore, such an approach has a strong bias toward classifyingnovel instances as one of the labeled classes.
In the following, we discuss our approach for learningto categorize all anchor boxes directly on the target ( unlabeled ) dataset in an end-to-end manner.
Supervised training on the source domain.
We start with a self-supervised backbone pretraining [26] on ImageNet [14] and train our object detection network on the source (labeled)domain using supervision for frequently occurring objects, such as 80 COCO classes (Fig.
2A, top ).
We use the labeled data to train the shared feature extractor (backbone) together with a class-agnosticregion proposal network (RPN), class-agnostic bounding box regression and segmentation heads, aswell as a classification head for categorization of labeled classes (Fig.
2A).
The features learned bythe backbone at this stage are trained for categorization of labeled objects to semantic classes and thenetwork is incentivized to learn representations that are agnostic to e.g. , color, object, orientation.
Discovery on the target domain.
During the following self-supervised (discovery) training phase onthe target domain we freeze all the layers apart from the primary (known) classification head, removeits background class, and attach a secondary (novel) classification head ( i.e. , categorization head)on top of RoI box features (Fig. 2B).
To obtain classification predictions at this phase, we pass RoIfeatures through both primary classification head and novel classification head, concatenating theirlogits.
We train the classification heads jointly using supervised classification and self-supervisedclassification losses.
The supervised classification loss effectively alleviates forgetting the weightsfor the known classes.
Different from standard supervised training, this loss is (i) evaluated based onlogits from both heads, and (ii) calculated only for positive region proposals, annotation-matched withsource dataset, excluding background proposals.
The self-supervised classification loss encouragesthe network to categorize every region proposal consistently under multiple transformations and aconstraint that the marginal distribution of predicted class assignments for the last batches follows aprior long-tail distribution (Fig.
2A, bottom ).
With such a joint training objective, we encourage ournetwork to learn to distinguish a large variety of semantic classes present in the regions sampled bythe RPN, instead of collapsing feature representations for unmatched regions into a background class,while retaining the capability to classify known objects.
Self-supervision via swapping cluster assignments.
To learn to discover novel classes we leverageclass-agnostic RoI proposals generated by RPN and build on self-labeling. We leverage clusteringmethods to generate cluster assignments (pseudo-labels) for each of the extracted object proposals inan online fashion and use the resulting labels to train the classification heads.
Leveraging standardclustering methods ( e.g. k-means [43]) to obtain such pseudo-labels was shown to result in unstabletraining [8, 9].
Instead, we follow [2, 1] and generate cluster assignments under a constraint that (i)the marginal probability distribution of generated assignments follows a specific prior distribution,and (ii) each proposal is classified consistently under multiple transformations.
We first extract RoI proposals and their features from the current batch using RPN and the sharedbackbone. We forward the resulting features through both known and novel classification heads toobtain logits for all classes. We then follow [2, 1] to generate pseudo-labels for each of the proposalsforcing the marginals of resulting current-batch cluster assignments to follow Gaussian distribution.
For that, following [2, 1] we optimize a cross-entropy loss between predicted class probabilities andcluster assignments w.r.t. the assignments.
This can be posed as a constrained optimization problemand solved as an optimal transport problem using the Sinkhorn-Knopp algorithm [13]. The resultingcluster assignments are then used to calculate cross-entropy loss. We further extend our approach and enforce the network to output consistent features and classificationpredictions for proposals under multiple augmentations, similarly to [10].
For that, for each imageafter generating the proposals coordinates, we first augment the image twice and only then extractthe features of the proposals for both of the image views.
This way we obtain two sets of featuresfor the same set of proposals. We generate pseudo-labels for each view (set of proposals’ features)individually, swap the resulting cluster assignments between views, and calculate the total loss as theaverage of the losses for both views.
Secondary classification head architecture.
As the box features are kept frozen during the discoveryphase, we add several non-linear projection layers on top of the box features.
This helps to learn tobetter disentangle feature representation of novel classes and aids the training of novel classificationhead, as shown experimentally in the ablations section (Sec. 4.2). We employ cosine classificationlayer, proposed in the context of few-shot learning [18, 12, 62] and related fields [10, 17, 7]. Wenote that the classification layer size directly implies the expected number of novel classes. As it isproblem-specific and requires analysis of desired granularity to match the labels of the target dataset,we perform an analysis of the sensitivity of our method to the layer size in ablations (Sec. 4.2).
Memory module.
Generating pseudo-labels of a good quality requires a large set of features torepresent the wide variety of object representations present in the data and having a small set offeatures is likely to result in noisy cluster assignments.
As we operate in scenarios with more thana thousand classes to discover, instead of using large batch sizes, we introduce a memory module[26, 10] to our network to store proposals’ features from the last batches.
For the Sinkhorn algorithm,at each iteration, we use features from both the memory module and current-batch proposals.
Fromthe resulting pseudo-labels, we use only those generated for the current-batch proposals for losscalculation and ignore the rest. For each view, we use a separate feature memory module.
Losses.
During the supervised phase we use a standard fully-supervised RCNN loss: loss sup =loss RPN + loss box + loss cls + loss mask , where each of the components corresponds to the RPNloss, localization branch loss, classification branch loss, and mask branch loss respectively [49, 25].
During the discovery phase, we keep a downscaled classification loss component and introduce aself-supervised loss to train both classification heads: loss disc =
loss ss + α · loss cls , where loss ssand α are the self-supervised loss and supervised loss scale coefficient respectively.
For the selfsupervised loss, we use cross-entropy loss on top of pseudo-labels generated online for the currentbatch of proposals.
We show in ablations that keeping the downscaled ( α < 1 ) supervised loss in theloop is beneficial and helps to avoid forgetting the known classes in the known-class classificationhead, while large α may create an unwanted bias towards known classes and harm learning of novelclasses.
Inference.
During the inference (Fig. 2B) we perform a standard forward RCNN pass [49, 25] basedon concatenated logits from both classification heads.
Experimental Evaluation
In this section, we discuss our evaluation test-bed, including datasets, evaluation settings, and metrics(Sec. 4.1).
In Sec.
4.2 we justify our main design decisions by studying NCDL performance in awell-controlled setting that closely follows a real-world scenario (training on a source labeled andtarget unlabeled dataset).
We then compare our method’s performance to several baselines (Sec. 4.3)and, finally, demonstrate that our method is applicable beyond our evaluation test-bed (Sec. 4.4).
4.1 Evaluation setting
COCO half + LVIS.
We re-purpose COCO 2017 [40] and LVIS v1 [22] datasets for running ablationsand comparisons with baselines and prior art.
We use annotations for 80 COCO classes during thesupervised training phase, aiming at further classifying additional 1000+ LVIS classes.
COCOdataset contains 123 K images with modal bounding box and segmentation mask annotations for 80classes.
The LVIS dataset contains a subset of 120 K COCO images with annotations for 1203 classesthat include all classes from COCO.
For our experiments, we follow LVIS training and validationsplits, resulting in 100 K training images and 20 K validation images.
To model the NCDL setup, wefurther split 100 K training images in half and use only 50 K images with 80 -class annotations from
COCO during the supervised training phase.
We refer to this dataset as COCO half .
We treat the restof 50 K images as additional unlabeled data used during the discovery phase.
LVIS + Visual Genome.
We perform a large-scale generalization experiment by using annotationsfor 1203 LVIS classes during the supervised learning ( i.e. , treating LVIS classes as labeled ), aimingto learn to discover extra 2700 + classes from the Visual Genome (VG)
v1.4 dataset [34] ( i.e. , treating
VG classes as unlabeled ).
We provide dataset details and splits (that ensure LVIS and VG classvocabularies do not overlap) in the supplementary.
We note that annotations provided in VG are notexhaustive per class and many of its classes are abstract or semantically overlapping both within VGand when compared to LVIS.
We thus focus on qualitative results, following the common practice,and provide quantitative results in supplementary.
Implementation details.
We base our model on Mask R- We check the results of our method as function of strength of the supervised loss.
[Table] [Table]
Evaluation metrics.
For quantitative evaluation, we follow [24, 23, 17, 63] to match predictedcluster assignments with annotated semantic classes.
We follow common practice and report meanaverage precision (mAP@[.5:.95]) [40].
We detail our evaluation procedure in the supplementary.
Baselines.
We compare our end-to-end trainable detector with state-of-the-art Weng et al.
[63]work, NCD methods ORCA [7], UNO[17], and k-means [43] baseline that operate on cropped objectinstances.
For NCD methods we use labeled instance annotations from COCO half dataset as labeledimages pool and cropped RPN proposals as unlabeled images pool.
For proposals extraction wetrain RCNN on COCO half under the same conditions as our base network, to ensure comparableevaluation conditions.
Having trained the NCD classifiers, we use the same RCNN to generate objectproposals for the validation images, apply the classifiers on top of the proposals, and continue with thestandard RCNN post-processing steps.
In addition, we report results of k-means clustering directlyapplied to FPN-based features.
We provide more details on baselines in the supplementary.
4.2 Model ablations
We ablate single and multi-phase training, the impact of keeping supervised loss in the loop, numberof novel categories, pseudo-labeling prior, model architecture decisions, and backbone pretraining.
In supplementary we provide further results on experiments with hyperparameters for proposalsextractor and how class-agnostic heads affect the performance of the supervised baseline.
Single-phase training and unfreezing R-CNN components during discovery.
As shown in Table3 we observe that training the network with both supervised and discovery losses from the beginning,summed up without any scaling, leads to divergence.
We hypothesize that this is due to noisygradients from our self-supervised signal, with R-CNN being very sensitive to the choice of lossesand hyperparameters [49].
To further support this claim, we experiment with a standard supervisedphase followed by a discovery phase without freezing any of the components.
We observe that such asetup also leads to divergence.
Unfreezing only the RoI heads during the discovery phase, however,does not lead to divergence, yet reduces the score by 1 . 32 mAP.
We thus keep all the componentsfrozen during the discovery phase, apart from known and novel classification heads.
The strength of the supervised loss.
We ablate the strength of the supervised classification lossthat is used along with the discovery loss during the discovery phase.
We vary the strength from0 (no supervised loss) to 1 (no downscaling) and provide the results in Table 1.
We observe thatexcluding supervised loss completely degrades the performance of the network for the known classesand benefits the performance of the novel classes.
This indicates that there is a tradeoff betweenperformance on known and novel classes, - keeping the full supervised loss introduces too much biastowards the former.
We consequently set the strength of the supervised loss to 0 .
5 in all experiments.
Sensitivity to the number of novel classes.
In Table 2 we experiment with varying the number ofnovel classes we use for the novel classification head.
We observe that the best number of classesfor the COCO + LVIS setup is 3000 , while other choices result in lower scores.
We conjecture thatsuch scores are sensitive to the semantic classes defined by the target dataset - with 3000 classes themodel learns the granularity and grouping that matches closest to the LVIS classes, but that does notnecessarily mean that a finer or coarser granularity categorization is worse.
In supplementary we alsoexperiment with attaching and training multiple heads jointly during the discovery phase and observethat scores slightly degrade in such setup but do not observe a significant drop in results.
Prior distribution for Sinkhorn clustering.
In Table 3 we experiment with using uniform priordistribution for class-marginals during pseudo-labels generation, as proposed in [9].
We observe [Table] [Table]
degradation of the score by 1 . 17 mAP, which confirms our hypothesis that a non-uniform priordistribution closer resembles a real-world long-tail class distribution.
Embedding projector and swapped assignments.
In Table 3 we demonstrate the effect of addingan MLP feature projector on top of RoI box features for novel heads classification.
We observe thatthe performance of the network improves by 1 . 62 mAP.
In the same table, we show that swappingpseudo-labels between proposals under multiple augmentations helps boost the score by 0 .
18 mAP.
Feature memory module.
We demonstrate the importance of memory module in Table 3.
We observe that removing the memory module significantly drops the performance by 4 . 09 mAP, especiallyfor the novel classes.
This highlights that for an efficient pseudo-labels generation it is crucial to usea set of features representative of the full feature space that we lack in the absence of memory.
Self-supervised pre-training matters.
We compare the results of our framework under differentbackbone initialization employed for the supervised pre-training stage.
For the randomly initializedbackbone we set all layers as trainable, while during the MoCo initialization the first two convolutionalblocks are kept frozen ( c.f. , [26]).
In Table 3 we observe that MoCo-initialized backbone improvesthe score by 2 . 15 mAP over randomly initialized backbone mode.
Proposals extractor hyperparameters.
We observe that when extracting the RPN proposals forself-supervised learning during the discovery phase the absence of NMS yields the best results andthe optimal number of proposals per image is 50.
We thus use such parameters for all the experiments.
We provide more details and ablation results in supplementary.
4.3 Comparison to prior work
In Table 4 we compare our method with k-means baselines and three recent state-of-the-art methodsthat we adapted to our scenario.
We present the results for both known (COCO) and unknown (the restof LVIS) classes.
Our method significantly outperforms the previous best NCD method UNO [17],by reaching 4 .
74 higher overall mAP.
We outperform UNO in both the known and especially novelclasses, where we reach 4 .
81 higher mAP.
We also significantly outperform approach by Weng etal.
Our method is the only method to reach over 1 mAP in novel classes, where we reach 5 . 42mAP.
We believe the main advantages of our method to be the memory module, critical for effectiveself-supervision, and a backbone trained in an end-to-end manner with the aid of class-agnosticlosses, further frozen during the discovery phase.
Further, our backbone benefits from high-levelsemantic features via FPN that is critical for the classification of small objects [39].
We providemore comparison details in the supplementary.
We note that as our method heavily relies on thegenerated region proposals, we are limited by the performance of region proposal network.
It alsodoes not update the backbone feature extractor, which could further boost the results, including theperformance of localization and segmentation components.
4.4 Cross-dataset Generalization
We analyze the generalization properties of our framework by evaluating the performance of ournetwork trained on LVIS and tested on VG datasets.
[Table]
We present quantitative results in Table 5. In Figure 4.4 we provide qualitative results with novelclasses discovered.
In Figure 4.4 we also visualize the 8 common discovered categories.
Uponmanual examination of the clusters, we observethat many of the novel classes are synonyms orhypernyms of known classes that appear in the LVIS dataset.
Conclusions
In this paper, we introduce an end-to-end RNCDL network for the task of novel class discovery,detection, and localization.
Our model is a two-stage object detection network that is able to classifyboth instances of the labeled, known classes and those of unlabeled, novel classes.
At the core of ourframework is a self-supervision guided by features of region proposals and a constraint for the novelclass assignments to follow a long-tail distribution.
In our experiments we demonstrate a significantimprovement over the previous state-of-the-art.
Furthermore, we demonstrate the ability to detectsemantic classes without any supervision at a large scale on the Visual Genome dataset.
Broader Impact
The success of deep learning, to a large extent, can be attributed to high-quality labeled datasets.
These datasets, for the most part, contain only a fixed number of categories (classes).
Deep learningmodels trained on such datasets can only make predictions for objects that belong to one of thetraining set categories.
However, the world consists of an extremely large number of categories,and enlarging the training set with more labeled data for all the categories is an infeasible problem.
Clearly, there is a need of shit towards solutions that are not limited by labeled data supervision.
Novel class category discovery has been mostly tackled in image classification setup, but there havebeen no works that give suitable solutions to the more challenging task of object detection.
In thiswork, we improve compared to previous methods by an order of magnitude.
While the problem is stillfar from solved, we hope that our method makes the first concrete step toward an effective solution,and would inspire others to tackle this challenging problem.
We expect that this method will lead tomore object detection works that aim to discover instances of novel classes.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[No] (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes]4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[Yes]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content
[N/A]5.
If crowdsourcing or conducted research with human subjects...
(a) Did
you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]