DetCLIP: Dictionary-Enriched Visual-Concept Paralleled Pre-training for Open-world Detection
Abstract
Open-world object detection, as a more general and challenging goal, aims torecognize and localize objects described by arbitrary category names.
Capitalized on large-scale Vision-Language (VL) pre-trained models (e.g., CLIP), someworks achieve promising open-world detection capability via distillation, which ishowever limited by costly feature extraction and the discrepancy between desiredregion-level features for detection and typical image-level features from VL models.
Recent GLIP instead reformulates this problem as a grounding problem byconcatenating all category names into a single sentence and gets pre-trained withthe sentence-region alignment scores, but it leads to clear inefficiency in interactionmodeling between category names and is restricted by the max length of the inputsentence.
This paper presents DetCLIP, a paralleled visual-concept pre-trainingmethod for open-world detection by resorting to knowledge enrichment from adesigned concept dictionary.
To achieve better learning efficiency, we first propose a novel paralleled concept formulation that extracts concepts separately andaligns entity concepts and image regions. This formulation also allows our modelto be pre-trained under different kinds of supervision (detection, grounding, andimage-text pairs).
We further design a concept dictionary (with descriptions) fromvarious online sources and detection datasets to form a unified concept space andprovide prior knowledge for each concept.
Based on the constructed dictionary,
DetCLIP can enrich the current concept with the concepts and their descriptionsexisting in the dictionary to facilitate open-domain learning. By sampling conceptsfrom the concept dictionary as negative samples for construction of the alignmentloss and the additional category inputs for label completion, our DetCLIP canfurther alleviate the partially labeled problem of the grounding and image-text pairdata. Experiments on large-scale open-world object detection benchmarks show
DetCLIP-T obtains around 9.9% mAP improvement over GLIP-T on LVIS withoutseeing any images in LVIS during pre-training The zero-shot transfer performanceof our DetCLIP-T even outperforms the fully-supervised model with the samebackbone, especially on rare classes (+13.5% on mAP).
Introduction
Most state-of-the-art object detection methods [37, 39, 3, 54] can only recognize and localize a predefined number of categories.
Their detection performance greatly relies on sufficient training data foreach category, which requires expensive and time-consuming human annotations, especially for therare classes that can only be distinguished from the expert.
Even though a great effort has been made,existing publicly available detection datasets only have a limited number of object categories, forinstance, 80 for COCO [31], 365 for Object365 [43], and 1203 for LVIS [16].
However, versatile openworld object detection is still out of reach mainly for two reasons: a)
Due to the long-tail problem [16],finding sufficient examples for rare categories is surprisingly challenging; b) Developing a largerdetection dataset requires extremely costly and labor-intensive manual annotation.
[Figure]
[Figure]
text pair Dataset
ACLIP
Training
Detecting All Classes
CommonClasses
RareClasses 1.
An australian shepherd herds cattle in a pen.
A sheepdog looks at three cows in a field.
A dog, three cows, and a woman enclosure.
A woman and a herding dog with three cattle.
A dog approaching cows grazing in a field. Cat
Computer
Sofa
A strikingblack and white bird with very long, thin red legs. On the
, image-text pair data are cheap and abundant on the Internet.
Recent visionlanguage (VL) pre-training methods (e.g., CLIP [36], ALIGN [26]) utilize those data to extend theiropen-domain capacity and have shown good zero-shot ability on various downstream classificationtasks.
Intuitively, some works [15, 51, 48, 12] try to extend a two-stage detector to an open-worlddetector by distilling the learned image embeddings of the cropped proposal regions
VL model. However, this paradigm requires costly feature from cropped images There also exists discrepancy between instance-level features and conventional image-level featuresextracted from VL models. [14] proposes a self-training pipeline with pseudo labels (i.e., nounphrases in caption) generated by a pre-trained VL model [26] while the phrases in caption are alwaystoo limited to cover all the objects in an image.
Moreover, their open-domain ability is determined by performance of the pre-trained VL models
Another line of work, GLIP [28] further proposes a grounding formulation of open-world detectionby directly utilizing both detection and grounding data for pre-training.
Specially for detectiondata, GLIP takes an image and a text prompt sentence that concatenates all the category names asinput (i.e., sequential formulation ).
A text encoder will output features for this sentence and then
GLIP aligns them with the region features extracted from the image encoder.
However, restrictedby the max length of the input token size of the text encoder, it is difficult for GLIP to operate on alarge number of categories or extend to more detailed description of the categories.
Furthermore, fullattention matrix upon all the categories has to be learned in the text encoder, which is unnecessaryand low-efficient for detection especially when the size of input categories increases.
To alleviate the above problems and further improve the open-domain ability, we present DetCLIP, adictionary-enriched visual-concept paralleled pre-training method for open-world detection .
Note that the “concepts” denotes the category names in detection data, and the phrases in groundingand image text pair data.
Specifically, we first design a novel paralleled concept formulationto improve learning efficiency.
Instead of feeding the whole prompt text sentence into the textencoder like GLIP, DetCLIP extracts each concept separately and parallelly feeds them into thetext encoder (i.e., paralleled formulation ).
This paralleled formulation allows the model to avoidunnecessary interaction between uncorrelated categories/phrases, and produce a longer descriptionfor each concept.
By converting detection data, grounding data, and image-text pair data intoparalleled formulation, DetCLIP can be pre-trained under different types of supervision to supportboth localization and open-domain capability.
[Figure]
During pre-training, existing datasets often have a largedomain gap and difference in their labeling space, e.g.,the same object concept with different names or hierarchical/inclusive structures of different concepts.
However, itis difficult to obtain these implicit relationships among concepts simply by their short names.
In order to form a moreunified concept space and provide prior knowledge (i.e.,implicit relationships) for each input concept, we proposea novel concept dictionary to enrich our prompt text concepts during joint pre-training, as shown in Fig.2.
Firstly, we construct the dictionary with conceptsextracted from online resources and existing large-scale detection datasets by considering both commonality and coherence.
Based on this concept dictionary, DetCLIP can automatically enrich thecurrent concept with the concepts and their descriptions existing in the dictionary to facilitate opendomain learning.
To alleviate the partial label problem for the grounding and image-text pair data,
DetCLIP further randomly samples concepts from the dictionary as negative samples to efficientlypre-train with the alignment loss.
Besides, all concepts in the dictionary are served as the additionalcategory inputs for label completion on the image-text pair data.
The proposed DetCLIP outperforms the state-of-the-art GLIP by large margins on multiple large-scaleopen-world detection benchmarks, including LVIS [16] with 1203 classes, and the other 13 detection datasets.
Particularly, DetCLIP-T achieves around 9.9% mAP improvement over GLIP-T on LVISwithout utilizing any images in LVIS during pre-training.
Compared to the baseline ATSS [52]model (with the same swin-T [33] backbone) trained on LVIS, our DetCLIP-T achieves 2.3% mAPimprovement in the total performance and 13.5% mAP improvement on the rare classes.
Related Work
Vision-Language Pre-training (VLP) .
Current Vision-Language Pre-training is a natural extensionand development of the successful pre-train-and-fine-tune scheme in the domains of natural languageprocessing (NLP)
[10, 1] and computer vision [11] community.
Dual-stream methods such as CLIP[36] and ALIGN [26] have shown great zero-shot classification ability by performing cross-modalcontrastive learning on large-scale image-text pairs from the Internet.
Single-stream approaches[27, 21] directly model the interaction between the visual and textual embeddings together by asingle transformer-based model, which can perform tasks such as image caption and VQA.
Recentapproaches such as VLMo [46] and BLIP [25] further explore a mixed architecture of single-streamand dual-stream models to enable a unified way of vision-language understanding and generation.
However, those approaches usually focus on whole-image representation learning and the pre-trainedmodels are usually designed for retrieval/generation tasks.
Current vision-language pre-trainingapproaches can not directly be applied to object detection task, i.e., a core computer vision task.
General Object Detection.
Object detection is a core problem in computer vision.
CNN-basedobject detection methods (one-stage detectors: YOLO [37], SSD [32] and ATSS [33] and two-stagedetectors: Faster R-CNN [39] and R-FCN [7]) usually use a classifier to map ROI (Region Of Interest)features into categories.
Further improvement of the CNN-based detectors such as K-Net [53] and
Panoptic-FCN [29] further introduce dynamic kernels and replace the static kernels in the convolutionlayers to improve the flexibility of models.
Recent transformer-based methods such as DETR [3]and Deformable DETR [54] try to formulate the object detection as a set prediction problem thatcan eliminate post-process NMS.
Those methods are constrained to predefined categories, whileour method tries to endow the detector with open-domain recognition ability which can detect anycategories by learning a wide range of concepts.
Zero-shot Object Detection/ Open-vocabulary Object Detection.
In the early setting of thisarea, zero-shot object detection aims to generalize the detector from known categories (training)to unknown categories (inference).
Under this setting, various works [5] try to find relationshipsbetween existing categories and unknown categories through pre-trained semantic/text features[44, 35, 38, 4, 13] knowledge graphs [41, 19, 49, 47] and so on.
However, the evaluation under thissetting is not general enough since people usually simply split the class name into known/unknowncategories in the single dataset and the transfer learning is still under similar domain.
On the otherhand, inspired by the success of vision-language~(VL) pre-training methods (e.g., CLIP [36]) andtheir good zero-shot ability, several methods attempt to perform zero-shot detection on a wider rangeof domains by leveraging a pre-trained VL model.
For example, [15] tries to distill the learned imageembeddings of the cropped proposal regions from CLIP [36]} to a student detector.
[14] proposesa self-training pipeline which utilizes Grad-CAM [42] and ALBEF [26].
However, those methodsare very slow because the feature extraction is repeated and the image-level representation may besub-optimal for the instance-wise tasks.
Recently, GLIP [28] and X-DETR [2] try to align region andlanguage features using a dot-product operation and can be trained end-to-end on both grounding dataand detection data.
Our method aims to design an open-domain detector that learns new conceptsefficiently and expands domain coverage from low-cost data from the Internet.
The Proposed Approach
This paper aims to develop a vision-language pre-training pipeline to enhance new concept learningfor open-world detection.
To achieve this goal, our DetCLIP leverages a new paralleled visionconcept per-training pipeline (Sec.3.1) for efficient training and a concept dictionary (Sec.3.2) toprovide external knowledge to automatically enrich the current input concept and alleviate the partiallabeling problem of the grounding and image-text pair data.
Our DetCLIP is pre-trained under hybridsupervision from detection data, grounding data and image-text pair data (Sec.3.3).
3.1 Paralleled Concept Formulation
A robust open-world detector is required to be trained with sufficient data covering enough visionconcepts.
Current available detection datasets lack enough concepts and have limited capacity due tothe costly annotation.
Leveraging data from other sources, e.g., grounding data and image-text pair [Figure] w 1 w 2 w 3 w 4 w 5 w Alignment Loss ww 1 w 2 w 3 w 4 w 5 w
Alignment
[Figure] w 1 c 1 c 2 c 3 c 4 c 5 c Alignment Loss c 1 c 2 c 3 c 4 c 5 c 6 w 2 w 3 w 5 w 6 w 1 w 4 w [Figure]
grounding
[Figure] data, is a feasible solution to augment the semantic coverage. To enable training with heterogeneoussupervisions, we are required to find a unified formulation for different formats of data. Fig.3 illustrates a comparison of the different concept formulation strategies that are used for pretraining with different types of data.
As shown in Fig.3(a)-(b), traditional detection and groundingtraining utilize different input formulations.
Detection treats each category as a fixed label and alignsregion features to the pre-defined label space, while the grounding training leverages the wholecaption sentence as input, models the attention between words, and adopts each token embedding inthe output embeddings for alignment.
To utilize both detection and grounding datasets for pre-training,
GLIP [28] converts the detection data to phrase grounding data, by replacing the object classificationlogits with the word-region alignment scores, which are associated with a sentence that concatenatesall the category names (see Fig.3(c)), i.e., the text input is [“person, bicycle, car, ... , toothbrush”].
We argue that this sequential form is not an effective formulation to model open-world object detectionas a vision-language task because it (1) leads to unnecessary interaction between category names inthe attention module; and (2) constraints the number of negative samples in contrastive learning dueto the limited context length of text input.
Ablation studies conducted for GLIP show that randomlyshuffling the word order in the grounding training data can even bring a slight improvement for thedownstream detection task, indicating that the noun phrase is more critical for the detection task,compared to the context information.
[Figure]
To address the above problem, our DetCLIP introducesa paralleled concept formulation to train with differentdata sources .
As shown in Fig.3(d), we extract the conceptnoun phrase for each bounding box and then feed them intothe text encoder individually to obtain the correspondingtext embedding.
In our paralleled design, context information is removed and model directly learns language featuresfrom each separate concepts, which is a more straightforward modeling for the detection task and improves the learning efficiency (see Fig.4).
Furthermore, our paralleled design can enable easy expansion of the augment of categorydescriptions (see Sec.3.2.2).
Detailed paralleled formulation for different types of data are as follows:
Detection data: Suppose there are k positive categories in an image, we first pad the category numberto N by randomly sampling the negative categories, where N is a predefined number of the conceptsfor constructing the alignment loss.
More specifically, the text input P can be formulated as { p n } Nn =1 ,where p n represents for the n -th category name, e.g.,
P =
[“person”, “bicycle”, “car”, ... , “toothbrush”].
Then, we feed N category names as separate sentences into the text encoder and use the embeddingof [end of sentence] token as the text embedding for each category.
The N text embeddings are thenconcatenated and matched with ground truth bounding boxes to construct the alignment loss.
Grounding data: We extract positive phrases for bounding boxes (provided by the groundingannotations), and drop other words in the caption.
To align the input format with detection data,we also pad the category number to N by sampling negative categories from our proposed conceptdictionary (see Sec.3.2.2).
An example of input P for grounding data can be:
[Equation] [Equation] herding dog”,
[Equation] cattle”, [Equation] , where neg 1 to neg m are sampled negative categories from the constructed concept dictionary.
Thensimilar steps with detection data can be conducted to construct the alignment loss.
Image-text pair data: Image-text pair data only contains images and the corresponding captions,while without any annotated bounding boxes.
To obtain object-level dense labels, we first use a pretrained class-agnostic region proposal network (RPN) to extract object proposals.
Then a powerfulpre-trained vision-language model like CLIP [36] or FILIP [50] is utilized to assign pseudo labels forthe proposals (see Appendix for more details.).
To tackle the concept-missing problem in the captionsof image-text pair data, instead of using noun phrases in the captions as the candidate categorynames following [28, 14], we propose to assign open-domain categories to the object proposals viaconstructing a large-scale concept dictionary (see Sec.3.2.2).
After obtaining object-level densepseudo labels, we convert the image-text pair data to the same format as the detection and groundingdata, and then a similar training procedure can be applied .
3.2 Concept Dictionary
In this work, we try to build an open-world object detector that can cover a wide range of concepts andcan be applicable to different types of datasets.
However, the existing detection/grounding/image-textpair datasets have a very large domain gap and difference in their labeling space.
For example, a boycan be annotated as “man”, “child” or “people” in the different datasets.
Moreover, there often existshierarchical/inclusive relationships between different concepts.
Knowing these implicit relationshipscan effectively facilitate the pre-training, however it is also clearly challenging to discover theserelationships with only a limited set of concept names.
Therefore, we propose to build a large-scale concept dictionary to form a unified concept spacefor different data sources and explicitly provide the useful relationships between various conceptsthrough definitions.
For example, the car is defined as “a motor vehicle with four wheels usuallypropelled by an internal combustion engine”, and the motorcycle is defined as “a motor vehicle withtwo wheels and a strong frame”.
By aggregating these definitions, we can conclude that the car andthe motorcycle are both motor vehicles with a difference in the number of wheels.
3.2.1 Constructing the Concept Dictionary
To construct a unified concept dictionary O , we collect concepts from multiple sources: (1) nounphrases extracted from the large-scale image-text pair dataset, i.e., YFCC100m; (2) category namesof existing public detection datasets (e.g., Objects365[43], OpenImages [24]); (3) object names fromthe manually-collected concept database, i.e., Things [18]).
For object names from the detectiondatasets and Things, we directly add them into the dictionary after deduplication.
For noun phrasesextracted from the YFCC, we filter out the concepts that appear less than a frequency of 100 orwithout definition in WordNet [34] to ensure the commonality.
Our concept dictionary O is thenconstructed by concatenating each word o l with its definition def l in WordNet: O = { o l : def l } Ll =1 ,where L is the number of concepts in our dictionary.
The constructed dictionary covers about 14kconcepts along with their definitions, and can be updated by directly adding new concepts andcorresponding definitions.
Based on the constructed concept dictionary, we then propose 2 techniquesutilizing it to boost the pre-training in the following section.
3.2.2 Knowledge Enrichment with Concept Dictionary
Concept Enrichment.
Based on the designed concept dictionary, we first retrieve the definitionfor each input concept to provide the prior knowledge (see Fig.5(b)).
During pre-training, for eachconcept p n in the training set, we can directly use its definition if p n is included in the dictionary
O .
If we cannot find a direct match in O , we will try to locate the most related concept in O bycalculating a similarity matrix S ′ ∈ R L .
The S ′ is calculated via the dot-product of the embeddingsfrom a pre-trained text encoder such as FILIP [50] with p n and all concept names { o l } Ll =1 as input.
Then we can find the most related concept { o l ∗ , where l ∗ =
argmax l ( S ′ ( l )) } in the dictionary, toretrieve an approximate definition def l ∗ .
The p n is then enriched with the retrieved definition andreformatted as { p ∗ n } = { p n , def l ∗ } .
An example of the enriched text input P ∗ = { p ∗ n } Nn =1 is:
[Figure]
P ∗ =
[“person, a human being.”, “bicycles, a wheeled vehicle that has two wheels and is moved byfoot pedals.”
, ... , “toothbrush, small brush has long handle used to clean teeth.”]
Partial Annotation Enrichment.
In the grounding or image-text pair data, only main objectsthat people care about are labeled in the caption, which is known as the partial labeling problem.
Compared with standard detection datasets which have sufficient positive and negative classes foreach image, pre-training with grounding and image-text pair datasets encounters two severe issues: 1)lack of annotations of negative concepts for learning discriminative concept embeddings; 2) lack ofannotations of partial positive concepts to efficiently train the model.
For the first problem, DetCLIPrandomly samples the concepts in the constructed dictionary O as the negative concepts toconstruct the alignment loss, instead of directly padding empty inputs (Fig.5(b)).
Note that since thenumber of concepts in the dictionary O is large (i.e., about 14k), the probability that the sampledconcepts are indeed in the image is extremely small.
For the second problem, to perform labelcompletion on image-text pair data during pseudo labeling, we add all the concepts in dictionary{ o l } Ll =1 as the additional category inputs, instead of using the original noun phrase in the caption tocalculate the similarity matrix.
Therefore, the concepts shown in the image while not in the captioncan also be labeled and then get pre-trained.
An illustration is also shown in Fig.7 to qualitativelyverify the effectiveness of label completion.
3.3 Model Architecture/Training Objective
As shown in Fig.5, the basic architecture of DetCLIP contains an image encoder Φ i to generatethe region features F I ∈ R M · D from the input image X , and a text encoder Φ t to obtain the textembeddings F T ∈ R N · D for the concepts in P ∗ , where M , N denote for the number of extractedregions and input concepts, respectively.
Then the alignment loss is constructed by calculating thealignment score S ∈ R N · M for all region-text pairs.
[Equation]
With the ground-truth alignment matrix G ∈ R N · M , the whole training objective L can be written as:
[Equation]
260 where L ALI , L CEN , and L REG denote the alignment loss, the centerness loss, and the regressionloss, respectively.
α and β represent the weight factor for L CEN and L REG , respectively.
Following
GLIP [28], we adopt the ATSS [52] detector as our image encoder.
We use the sigmoid focal loss [30]for L ALI , the sigmoid
Experimental Results
Implementation Details.
We pre-train all the models based on Swin-Transformer [33] backboneswith 32 GPUs.
AdamW optimizer [22] is adopted and batch size is set to 128.
The learning rateis set to 2.8x10 − 4 for the parameters of the visual backbone and detection head, and 2.8x10 − 5 forthe language backbone.
Without otherwise specified, all models are trained with 12 epochs and thelearning rate is decayed with a factor of 0.1 at the 8-th and the 11-th epoch.
The max token length for [Table] each input sentence is set to 48.
The number of the concepts N in text input P is set to 150 and thenumber of region features M is determined by the feature map size and the number of pre-definedanchors.
The loss weight factors α and β are both set to 1.0.
The training of DetCLIP-T with Swin-Tbackbone tasks 63 hours when 32 GPUs are used.
MMDetection [6] code-base is used.
Training Data.
Our model is trained with hybrid supervision from different kinds of data, i.e.,detection data, grounding data, and image-text pair data.
More specifically, for detection data, weuse a sampled Objects365 V2 [43] dataset (denoted as O365 in the following sections) with 0.66Mtraining images.
Here we do not use the whole dataset since training with sampled data is moreefficient and suffices to demonstrate the effectiveness of our method.
For grounding data, we usegold grounding data (denoted as GoldG) introduced by MDETR [20].
Moreover, following GLIP[28], we remove the training samples contained in LVIS [16] dataset for fair zero-transfer evaluation,which results in 0.77M training data.
For image-text pair data, we perform object-level dense pseudolabeling on YFCC100m [45] dataset with a pre-trained CLIP [36] model, and sample a subset of 1Mtraining images from the results containing objects with similarities scores above a given threshold.
Finally, our training set contains a total of 2.43M images.
Compared to GLIP’s 27M training data,
DetCLIP only uses less than 10% data, but achieves better results .
More details are in Appendix.
Benchmark Settings.
We evaluate our method mainly on LVIS [16] which contains 1203 categories.
Following GLIP [28] and MDETR [20], we evaluate on the 5k minival subset and report the zero-shotfixed AP [9] for a fair comparison.
We do not focus the performance on COCO [31] since it onlycontains 80 common categories that are fully covered by the training dataset Objects365 [43], whichmay not sufficient to reflect generalization ability of a model in the open-domain detection setting.
To further study the generalization ability of our method, following GLIP [28], we also evaluate theaveraged AP on other 13 downstream detection datasets published on Roboflow 1 .
4.1 Open-world Detection Results
We train our DetCLIP with two backbones, i.e., swin-T [33] and swin-L. Distinct from GLIP [28],which introduces additional heavy modules like Dynamic Head [8] and cross-modal fusion, wedirectly adopt the vanilla ATSS [52] as our vision encoder to keep our architecture as neat as possible.
Following GLIP, for Swin-T backbone, we also train 3 versions of models, i.e., DetCLIP-T(A),
DetCLIP-T(B) and a complete version DetCLIP-T, which differ in using different training data.
Table 1 reports the results on LVIS [16].
Results with LVIS as pre-training data stand for the fullysupervised models trained with annotated data.
With our proposed paralleled formulation, introducingmore training data from different sources can consistently improve the performance.
I.e., comparing
DetCLIP-T(A) trained with only detection data with DetCLIP-T trained with additional groundingand image-text pair data, we can observe a considerable performance gain (28.8% AP v.s. 35.9%
AP).
Besides, befitting from the proposed effective framework, our DetCLIP models outperform their [Table]
[Figure] DetCLIP-T(A) (resp.
DetCLIP-T) surpasses GLIP-T(A)(resp.
GLIP-T) by 10.3% (resp.
Note that our models are more lightweight (without heavy DyHead and cross-modal fusion) and trained with fewer epochs (our 12 vs. GLIP’s 24) and much lessdata.
Notably, our DetCLIP-T’s zero-shot performance even beats the fully-supervised modelwith the same backbone by utilizing weak-annotated data like image-text pair data.
Results on LVISfull validation dataset and 13 detection datasets can refer to the Appendix.
[Table]
Efficiency Comparison.
To demonstrate the [Table] efficiency of our proposed DetCLIP, we directlycompare the training and inference speed of DetCLIP-T with the GLIP-T [28] in Table 2.
With the same setting of training with 32 {+V
+} GPUs, the total training time for GLIP-T isabout 10.7K GPU hours (5X than us) due to its heavy backbone and more image-text pair trainingdata.
On the other hand, DetCLIP-T achieves the [-2.
-]2.3 FPS (0.43 s/image) on a single V100 whenperforming inference on LVIS [16], while GLIP-T can only achieve 0.12 FPS (8.6 s/image).
Withmuch better training and inference efficiency, DetCLIP-T can still outperform GLIP-T 9.9% on LVIS.
Qualitative Visualizations.
We illustrate the bounding box predictions on LVIS [16] dataset from
DetCLIP-T and GLIP-T [28] model in Fig.6.
We can observe that by adopting the paralleled conceptformulation and the external knowledge from the concept dictionary, our DetCLIP can outperform
GLIP both on the accuracy and completeness of the predicted labels, especially on the rare classes.
4.2 Ablation Studies
Table 3 studies the effectiveness of two core components of DetCLIP, i.e., the paralleled formulationand the knowledge enrichment with concept dictionary.
The first row stands for our implementationof a GLIP-A [28] model, which is modeled with sequential formulation and uses only detection datafor training.
Due to the implementation discrepancy, our version can achieve 23.7% zero-shot AP on
LVIS [16], which is higher than the official’s 18.5% and serves as a stronger baseline.
First, applying the paralleled formulation can bring significant improvements (row 2), boostingthe performance to 27.8%.
This indicates that paralleled formulation is much more effective than [Figure] [Figure] sequential formulation for modeling object detection as a vision-language task.
However, directlyapplying the same approach to a larger scale dataset with heterogeneous supervisions, e.g., detectionplus grounding, can hurt the performance (row 4).
We speculate this is because paralleled formulationweakens the interaction between text concepts, resulting in the model’s inability to effectively construct the connections between semantic-related concepts.
Therefore, we introduce word definitions tothe class names to help bridge relationships between different concepts, which boosts the performanceto 32.2% (row 5).
Sampling negative categories from the concept dictionary also helps better utilizegrounding data, improving the performance to 34.4% (row 7).
Further introducing image-text pairdatasets like YFCC [45] can bring substantial improvement for rare categories (row 8), while utilizingconcept dictionary for label completion during pseudo labeling finally improves the overall AP to35.9% (row 9).
A similar performance pattern is also observed on 13 downstream detection datasets.
Pseudo labels with concept dictionary.
Fig.7 visualizes the results of label completion via conceptdictionary.
Multiple effectiveness can be observed.
E.g., cases (a) and (b) demonstrate conceptdictionary contributes to labeling bbox with the category names not shown in the captions.
In (c),a finer-grained pseudo label can be produced, i.e., ‘antheraea polyphemus’ v.s. the original ‘brownbutterfly’, which helps the learning of rare categories.
In (d), label noise in the caption is alleviated.
Importance of concept enrichment.
To directly illustrate how the category definition helps themodel achieve better detection performance, we conduct the experiments to infer with differenttext inputs with DetCLIP-T utilized.
We compare the three different cases, i.e., no definition, truedefinition and false definition (use definition of another category), and report the alignment score fordifferent objects in Fig.8.
Observations are: (1) True definition helps the model to better determinethe category (i.e., increases the confidence score); (2) Wrong definition may confuse the model andbring the false positive samples; (3) Both the category name and definition matter to the model.
Conclusion
In this paper, we propose a novel open-world detection pre-training framework named DetCLIP,aiming at improving the open-domain ability and learning efficiency of the open-world detector.
Byunifying three kinds of supervision via paralleled concept formulation, our DetCLIP can learn fromdifferent domains and enhance the training efficiency.
We further propose a concept dictionary moduleto improve the discovery and coverage of the novel knowledge by importing external knowledge.
Theproposed usages of the concept dictionary achieves better open-world detection result in terms of bothcommon and rare categories on LVIS.
Experiments on multiple downstream detection datasets suggestthat our DetCLIP is more powerful than current SOTA open-world detectors such as GLIP [28]. Checklist For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See the Appendix
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] See the
Appendix(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]2.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main exper-
imental results (either in the supplemental material or as a URL)?
[Yes]
We already
include the data and instructions in Section 4, and the code will be released upon
acceptance.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes] See Section 4 and Appendix.
(c) Did you report error bars (e
g., with respect to the random seed after running experiments multiple times)?
[N/A] Our experiments are quite stable with multiple runs.
All
results are observed via a fixed seed.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See Section 4.
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[Yes] All the used existing assets areopen-source.
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
[Table]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
[Table]
using/curating?
[Yes] We use open source data and the data is anonymized.
[Table]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
[Table]
information or offensive content?
[Yes] We use open source data and the data is
[Table]
anonymized.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
[Table]
spent on participant compensation?
[N/A]