O PEN - VOCABULARY O BJECT D ETECTION VIA V ISION AND L ANGUAGE K NOWLEDGE D ISTILLATION
A BSTRACT
We aim at advancing open-vocabulary object detection, which detects objects described by arbitrary text inputs.
The fundamental challenge is the availability of training data.
Existing object detection datasets only contain hundreds of categories, and it is costly to scale up further.
To overcome this challenge, we propose ViLD , a training method via Vi sion and L anguage knowledge D istillation.
Our method distills the knowledge from a pretrained open-vocabulary image classiﬁcation model (teacher) into a two-stage detector (student).
Speciﬁcally, we use the teacher model to encode category texts and image regions of object proposals.
Then we train a student detector, whose region embeddings of detected boxes are aligned with the text and image embeddings inferred by the teacher.
We benchmark on LVIS by holding out all rare categories as novel categories not seen during training.
ViLD obtains 16.1 mask AP r , even outperforming the supervised counterpart by 3.8 with a ResNet-50 backbone.
The model can directly transfer to other datasets without ﬁnetuning, achieving 72.2 AP 50 , 36.6 AP and 11.8 AP on PASCAL VOC, COCO and Objects365, respectively.
On COCO, ViLD outperforms previous SOTA (Zareian et al., 2021) by 4.8 on novel AP and 11.4 on overall AP.
Code and demo are open-sourced 1 .
I NTRODUCTION
Consider Fig.
1, can we design object detectors beyond recognizing only base categories ( e.g ., toy) present in training labels and expand the vocabulary to detect novel categories ( e.g ., toy elephant)?
In this paper, we aim to train an open-vocabulary object detector that detects objects in any novel categories described by text inputs, using only detection annotations in base categories.
Existing object detection algorithms often learn to detect only the categories present in detection datasets.
A common approach to increase the detection vocabulary is by collecting images with more labeled categories.
The research community has recently collected new object detection datasets with large vocabularies (Gupta et al., 2019; Kuznetsova et al., 2020).
LVIS (Gupta et al., 2019) is a milestone of these efforts by building a dataset with 1,203 categories.
With such a rich vocabulary, it becomes quite challenging to collect enough training examples for all categories.
By Zipf’s law, object categories naturally follow a long-tailed distribution.
To ﬁnd sufﬁcient training examples for rare categories, signiﬁcantly more data is needed (Gupta et al., 2019), which makes it expensive to scale up detection vocabularies.
On the other hand, paired image-text data are abundant on the Internet.
Recently, Radford et al.
(2021) train a joint vision and language model using 400 million image-text pairs and demonstrate impressive results on directly transferring to over 30 classiﬁcation datasets.
The pretrained text encoder is the key to the zero-shot transfer ability to arbitrary text categories.
Despite the great success on learning image-level representations, learning object-level representations for open-vocabulary detection is still challenging.
In this work, we consider borrowing the knowledge from a pretrained open-vocabulary classiﬁcation model to enable open-vocabulary detection.
[Figure]
We begin with an R-CNN (Girshick et al., 2014) style approach.
We turn open-vocabulary detection into two sub-problems: 1) generalized object proposal and 2) open-vocabulary image classiﬁcation.
We train a region proposal model using examples in the base categories.
Then we use the pretrained open-vocabulary image classiﬁcation model to classify cropped object proposals, which can contain both base and novel categories.
We benchmark on LVIS (Gupta et al., 2019) by holding out all rare categories as novel categories and treat others as base categories.
To our surprise, the performance on the novel categories already surpasses its supervised counterpart.
However, this approach is very slow for inference, because it feeds object proposals one-by-one into the classiﬁcation model.
To address the above issue, we propose ViLD ( Vi sion and L anguage knowledge D istillation) for training two-stage open-vocabulary detectors.
ViLD consists of two components: learning with text embeddings (ViLD-text) and image embeddings (ViLD-image) inferred by an open-vocabulary image classiﬁcation model, e.g ., CLIP.
In ViLD-text , we obtain the text embeddings by feeding category names into the pretrained text encoder.
Then the inferred text embeddings are used to classify detected regions.
Similar approaches have been used in prior detection works (Bansal et al., 2018;
Rahman et al., 2018; Zareian et al., 2021).
We ﬁnd text embeddings learned jointly with visual data can better encode the visual similarity between concepts, compared to text embeddings learned from a language corpus, e.g ., GloVe (Pennington et al., 2014).
Using CLIP text embeddings achieves 10.1 AP r (AP of novel categories) on LVIS, signiﬁcantly outperforming the 3.0 AP r of using GloVe.
In ViLD-image , we obtain the image embeddings by feeding the object proposals into the pretrained image encoder.
Then we train an object detector whose region embeddings of detected boxes are aligned with these image embeddings.
In contrast to ViLD-text, ViLD-image distills knowledge from both base and novel categories since the proposal network generalizes well, while ViLD-text only learns from base category concepts.
Distillation allows ViLD a generality in the choice of teacher and student architectures. ViLD is also energy-efﬁcient as it works with offthe-shelf open-vocabulary image classiﬁers. We experiment with the CLIP and ALIGN (Jia et al., 2021) teacher models with different architectures (ViT and EfﬁcientNet).
We show that ViLD, which jointly learns from text and image embeddings, achieves 16.1 AP r on LVIS, surpassing the supervised counterpart by 3.8 .
To push the performance limit, we use ALIGN as a stronger teacher model and attain the best performance of 26.3 AP r , which is only 3.7 AP r worse than the fully-supervised 2020 LVIS Challenge winner (Tan et al., 2020) that uses additional tricks.
We also attain SOTA results on COCO (Lin et al., 2014), outperforming the previous SOTA (Zareian et al., 2021) by 4.8 novel AP and 11.4 overall AP.
In addition, ViLD trained on LVIS can directly transfer to other detection datasets including Objects365, COCO, PASCAL VOC.
R ELATED W ORK
Increasing vocabulary in visual recognition: Recognizing objects using a large vocabulary is a long-standing research problem in computer vision.
One focus is zero-shot recognition, aiming at recognizing categories not present in the training set.
Early works (Farhadi et al., 2009;
Rohrbach et al., 2011; Jayaraman & Grauman, 2014) use visual attributes to create a binary codebook representing categories, which is used to transfer learned knowledge to unseen categories.
In this direction, researchers have also explored class hierarchy, class similarity, and object parts as discriminative features to aid the knowledge transfer (Rohrbach et al., 2011; Akata et al., 2016;
Zhao et al., 2017; Elhoseiny et al., 2017; Ji et al., 2018;
Cacheux et al., 2019; Xie et al., 2020).
Another focus is learning to align latent image-text embeddings, which allows to classify images using arbitrary texts.
Frome et al.
and Norouzi et al.
(2014) are pioneering works that learn a visual-semantic embedding space using deep learning.
Wang et al.
(2018) distills information
[Figure] [Figure]
R
[Figure]
2 ... R 1 ·B n R 1 ·N 1 ... R
[Figure]
k R
[Figure]
·B 2 ..
[Figure]
R 2 ·B
[Figure] [Figure]
R 2 ·N
[Figure]
from both word embeddings and knowledge graphs. Recent work CLIP (Radford et al., 2021) and
ALIGN (Jia et al., 2021) push the limit by collecting million-scale image-text pairs and then training joint image-text models using contrastive learning.
These models can directly transfer to a suite of classiﬁcation datasets and achieve impressive performances.
While these work focus on image-level open-vocabulary recognition, we focus on detecting objects using arbitrary text inputs.
Increasing vocabulary in object detection: It’s expensive to scale up the data collection for large vocabulary object detection.
Zhao et al.
(2020) and Zhou et al.
(2021) unify the label space from multiple datasets.
Joseph et al.
(2021) incrementally learn identiﬁed unknown categories.
Zero-shot detection (ZSD) offers another direction.
Most ZSD methods align region features to pretrained text embeddings in base categories (Bansal et al., 2018;
Demirel et al., 2018; Rahman et al., 2019;
Hayat et al., 2020; Zheng et al., 2020).
However, there is a large performance gap to supervised counterparts.
To address this issue, Zareian et al.
(2021) pretrain the backbone model using image captions and ﬁnetune the pretrained model with detection datasets.
In contrast, we use an image-text pretrained model as a teacher model to supervise student object detectors.
All previous methods are only evaluated on tens of categories, while we are the ﬁrst to evaluate on more than 1,000 categories.
M ETHOD
Notations: We divide categories in a detection dataset into the base and novel subsets, and denote them by C B and C N .
Only annotations in C B are used for training.
We use T ( · ) to denote the text encoder and V ( · ) to denote the image encoder in the pretrained open-vocabulary image classiﬁer.
3. L OCALIZATION FOR NOVEL CATEGORIES
The ﬁrst challenge for open-vocabulary detection is to localize novel objects.
We modify a standard two-stage object detector, e.g ., Mask R-CNN (He et al., 2017), for this purpose.
We replace its classspeciﬁc localization modules, i.e ., the second-stage bounding box regression and mask prediction layers, with class-agnostic modules for general object proposals.
For each region of interest, these modules only predict a single bounding box and a single mask for all categories, instead of one prediction per category.
The class-agnostic modules can generalize to novel objects.
3. O PEN - VOCABULARY DETECTION WITH CROPPED REGIONS
Once object candidates are localized, we propose to reuse a pretrained open-vocabulary image classiﬁer to classify each region for detection.
[Figure]
Image embeddings: We train a proposal network on base categories C B and extract the region proposals ˜ r ∈ (cid:101) P ofﬂine.
We crop and resize proposals, and feed them into the pretrained image encoder V to compute image embeddings V ( crop ( I, ˜ r )) , where I is the image.
We ensemble the image embeddings from 1 × and 1 .
5 × crops, as the 1 . 5 × crop provides more context cues.
The ensembled embedding is then renormalized to unit norm:
V ( crop ( I, ˜ r { 1 × , 1 . 5 ×} ))
= v (cid:107) v (cid:107) , where v = V ( crop ( I, ˜ r 1 × ))
+ V ( crop ( I, ˜ r 1 .
5 × )) .
[Equation]
Text embeddings: We generate the text embeddings ofﬂine by feeding the category texts with prompt templates, e.g ., “a photo of { category } in the scene”, into the text encoder T .
We ensemble multiple prompt templates and the synonyms if provided.
Then, we compute cosine similarities between the image and text embeddings.
A softmax activation is applied, followed by a per-class NMS to obtain ﬁnal detections.
The inference is slow since every cropped region is fed into V .
3. V I LD: V ISION AND L ANGUAGE KNOWLEDGE D ISTILLATION .
We propose ViLD to address the slow inference speed of the above method.
ViLD learns region embeddings in a two-stage detector to represent each proposal r .
We denote region embeddings by
R ( φ ( I ) , r ) , where φ ( · ) is a backbone model and R ( · ) is a lightweight head that generates region embeddings.
Speciﬁcally, we take outputs before the classiﬁcation layer as region embeddings.
Replacing classiﬁer with text embeddings: We ﬁrst introduce ViLD-text .
Our goal is to train the region embeddings such that they can be classiﬁed by text embeddings.
Fig.
3(b) shows the architecture and training objective.
ViLD-text replaces the learnable classiﬁer in Fig. 3(a) with the text embeddings introduced in Sec. 3.2.
Only T ( C B ) , the text embeddings of C B , are used for training.
For the proposals that do not match to any groundtruth in C B , they are assigned to the background category.
Since the text “background” does not well represent these unmatched proposals, we allow the background category to learn its own embedding e bg .
We compute the cosine similarity between each region embedding R ( φ ( I ) , r ) and all category embeddings, including T ( C B ) and e bg .
Then we apply softmax activation with a temperature τ to compute the cross entropy loss.
To train the ﬁrst-stage region proposal network of the two-stage detector, we extract its region proposals r ∈ P online, and train the detector with ViLD-text from scratch.
The loss for ViLD-text can be written as:
[Equation]
where sim ( a , b ) = a (cid:62) b / ( (cid:107) a (cid:107)(cid:107) b (cid:107) ) , t i denotes elements in T ( C B ) , y r denotes the class label of region r , N is the number of proposals | P | per image, and L CE is the cross entropy loss.
During inference, we include novel categories ( C N ) and generate T ( C B ∪ C N ) (sometimes T ( C N ) only) for open-vocabulary detection (Fig. 2).
Our hope is that the model learned from annotations in C B can generalize to novel categories C N .
Distilling image embeddings: We then introduce ViLD-image , which aims to distill the knowledge from the teacher image encoder V into the student detector.
Speciﬁcally, we align region embeddings R ( φ ( I ) , ˜ r ) to image embeddings V ( crop ( I, ˜ r )) introduced in Sec. 3.2.
To make the training more efﬁcient, we extract M proposals ˜ r ∈ ˜ P ofﬂine for each training image, and precompute the M image embeddings.
These proposals can contain objects in both C B and
C N , as the network can generalize.
In contrast, ViLD-text can only learn from C B .
We apply an Lloss between the region and image embeddings to minimize their distance.
The ensembled image embeddings in Sec.
3.2 are used for distillation:
[Equation]
Fig. 3(c) shows the architecture.
Zhu et al.
(2019) use a similar approach to make Faster R-CNN features mimic R-CNN features, however, the details and goals are different: They reduce redundant context to improve supervised detection; while ViLD-image is to enable open-vocabulary detection on novel categories.
The total training loss of ViLD is simply a weighted sum of both objectives:
[Equation]
where w is a hyperparameter weight for distilling the image embeddings.
Fig. 3(d) shows the model architecture and training objectives.
ViLD-image distillation only happens in training time.
During inference, ViLD-image, ViLD-text and ViLD employ the same set of text embeddings as the detection classiﬁer, and use the same architecture for open-vocabulary detection (Fig. 2).
3. M ODEL ENSEMBLING
In this section, we explore model ensembling for the best detection performance over base and novel categories.
First, we combine the predictions of a ViLD-text detector with the open-vocabulary image classiﬁcation model.
The intuition is that ViLD-image learns to approximate the predictions of its teacher model, and therefore, we assume using the teacher model directly may improve performance.
We use a trained ViLD-text detector to obtain top k candidate regions and their conﬁdence scores.
Let p i, ViLD-text denote the conﬁdence score of proposal ˜ r belonging to category i .
We then feed crop ( I, ˜ r ) to the open-vocabulary classiﬁcation model to obtain the teacher’s conﬁdence score p i, cls .
Since we know the two models have different performance on base and novel categories, we introduce a weighted geometric average for the ensemble:
[Equation]
λ is set to 2 / 3 , which weighs the prediction of ViLD-text more on base categories and vice versa.
Note this approach has a similar slow inference speed as the method in Sec. 3.2.
Next, we introduce a different ensembling approach to mitigate the above inference speed issue.
Besides, in ViLD, the cross entropy loss of ViLD-text and the L 1 distillation loss of ViLD-image is applied to the same set of region embeddings, which may cause contentions.
Here, instead, we learn two sets of embeddings for ViLD-text (Eq. 2) and ViLD-image (Eq. 3) respectively, with two separate heads of identical architectures.
Text embeddings are applied to these two regions embeddings to obtain conﬁdence scores p i, ViLD-text and p i, ViLD-image , which are then ensembled in the same way as Eq. 5, with p i, ViLD-image replacing p i, cls .
We name this approach ViLD-ensemble .
E XPERIMENTS
Implementation details: We benchmark on the Mask R-CNN (He et al., 2017) with ResNet (He et al., 2016)
FPN (Lin et al., 2017) backbone and use the same settings for all models unless explicitly speciﬁed.
The models use 1024 × 1024 as input image size, large-scale jittering augmentation of range [0 . 1 , 2 . 0] , synchronized batch normalization (Ioffe & Szegedy, 2015; Girshick et al., 2018) of batch size 256, weight decay of 4e-5, and an initial learning rate of 0.32.
We train the model from scratch for 180,000 iterations, and divide the learning rate by 10 at 0.9 × , 0.95 × , and 0.975 × of total iterations.
We use the publicly available pretrained CLIP model 2 as the open-vocabulary classiﬁcation model, with an input size of 224 × 224.
The temperature τ is set to 0.01, and the maximum number of detections per image is 300.
We refer the readers to Appendix D for more details.
4. B ENCHMARK SETTINGS
We mainly evaluate on LVIS (Gupta et al., 2019) with our new setting.
To compare with previous methods, we also use the setting in Zareian et al.
(2021), which is adopted in many zero-shot detection works.
LVIS:
We benchmark on LVIS v1.
LVIS contains a large and diverse set of vocabulary (1203 categories) that is more suitable for open-vocabulary detection.
We take its 866 frequent and common categories as the base categories C B , and hold out the 337 rare categories as the novel categories
C N .
AP r , the AP of rare categories, is the main metric.
COCO: Bansal et al.
(2018) divide COCO-2017 (Lin et al., 2014) into 48 base categories andnovel categories, removing 15 categories without a synset in WordNet hierarchy.
We follow previous works and do not compute instance masks.
We evaluate on the generalized setting.
4. L EARNING GENERALIZABLE OBJECT PROPOSALS
We ﬁrst study whether a detector can localize novel categories when only trained on base categories.
We evaluate the region proposal networks in Mask R-CNN with a ResNet-50 backbone.
Tableshows the average recall (AR)
Lin et al.
(2014) on novel categories.
Training with only base categories performs slightly worse by ∼ 2 AR at 100, 300, and 1000 proposals, compared to using both base and novel categories.
This experiment demonstrates that, without seeing novel categories during training, region proposal networks can generalize to novel categories, only suffering a small performance drop.
We believe better proposal networks focusing on unseen category generalization should further improve the performance, and leave this for future research.
[Table]
4. O PEN - VOCABULARY CLASSIFIER (CLIP) ON CROPPED REGIONS
In table 2, we evaluate the approach in Sec. 3.2, i.e ., using an open-vocabulary classiﬁer to classify cropped region proposals.
We use CLIP in this experiment and ﬁnd it tends to output conﬁdence scores regardless of the localization quality (Appendix B).
Given that, we ensemble the CLIP conﬁdence score with a proposal objectness score by geometric mean.
Results show it improves both base and novel APs.
We compare with supervised baselines trained on base/base+novel categories, as well as Supervised-RFS (Mahajan et al., 2018; Gupta et al., 2019) that uses category frequency for balanced sampling.
CLIP on cropped regions already outperforms supervised baselines on AP r by a large margin, without accessing detection annotations in novel categories.
However, the performances of AP c and AP f are still trailing behind.
This experiment shows that a strong openvocabulary classiﬁcation model can be a powerful teacher model for detecting novel objects, yet there is still much improvement space for inference speed and overall AP.
[Table] [Table]
4. V ISION AND LANGUAGE KNOWLEDGE DISTILLATION
We evaluate the performance of ViLD and its variants (ViLD-text, ViLD-image, and ViLDensemble), which are signiﬁcantly faster compared to the method in Sec. 4.3.
Finally, we replace
CLIP with ALIGN to demonstrate our best performance.
Table 3 summarizes the results.
Text embeddings as classiﬁers (ViLD-text): We evaluate ViLD-text using text embeddings generated by CLIP, and compare it with GloVe text embeddings (Pennington et al., 2014) pretrained on a large-scale text-only corpus.
Table 3 shows ViLD-text achieves 10.1 AP r , which is signiﬁcantly better than 3.0 AP r using GloVe.
This demonstrates the importance of using text embeddings that are jointly trained with images.
ViLD-text achieves much higher AP c and AP f compared to CLIP on cropped regions (Sec. 4.3), because ViLD-text uses annotations in C B to align region embeddings with text embeddings.
The AP r is worse, showing that only using 866 base categories in LVIS does not generalize as well as CLIP to novel categories.
Distilling image embeddings (ViLD-image):
We use the model in Section 4.3 as the teacher model and distill from its image embeddings of proposals, inferred by CLIP’s image encoder, with a distillation weight of 1.0.
Experiments show that ensembling with objectness scores doesn’t help with other ViLD variants, so we only apply it to ViLD-image.
Ideally, the detection performance of ViLD-image (student) and CLIP on cropped regions (teacher) would be similar.
As shown in
Table 3, the performance gap is 7.7 AP r .
This demonstrates that visual distillation works for openvocabulary detection but yields a decrease in performance.
Text+visual embeddings (ViLD): ViLD shows the beneﬁts of combining distillation loss (ViLDimage) with classiﬁcation loss using text embeddings (ViLD-text).
We explore different hyperparameter settings in Appendix Table 7 and observe a consistent trade-off between AP r to AP c,f , which suggests there is a competition between ViLD-text and ViLD-image.
In Table 3, we compare
ViLD with other methods.
Its AP r is 6.0 higher than ViLD-text and 4.9 higher than ViLD-image, indicating combining the two learning objectives boosts the performance on novel categories.
ViLD outperforms Supervised-RFS by 3.8 AP r , showing our open-vocabulary detection approach is better than supervised models on rare categories.
[Table]
Model ensembling:
We study methods discussed in Sec. 3.4 to reconcile the conﬂict of joint training with ViLD-text and ViLD-image.
We use two ensembling approaches: 1) ensembling ViLD-text with CLIP ( ViLD-text+CLIP ); 2) ensembling ViLD-text and ViLD-image using separate heads ( ViLD-ensemble ).
As shown in Table 3, ViLD-ensemble improves performance over ViLD, mainly on AP c and AP r .
This shows ensembling reduces the competition.
ViLD-text+CLIP obtains much higher AP r , outperforming ViLD by 6.
[Table]
and maintains good AP c,f .
Note that it is slow and impractical for real world applications.
This experiment is designed for showing the potential of using open-vocabulary classiﬁcation model for open-vocabulary detection.
Stronger teacher model: We use CLIP ViT-L/14 and ALIGN (Jia et al., 2021) to explore the performance gain with a stronger teacher model (details in Appendix D).
As shown in Table 3, both models achieve superior results compared with R50 ViLD w/ CLIP.
The detector distilled from
ALIGN is only trailing to the fully-supervised 2020 Challenge winner (Tan et al., 2020) by 3.7 AP r , which employs two-stage training, self-training, and multi-scale testing etc .
The results demonstrate
ViLD scales well with the teacher model, and is a promising open-vocabulary detection approach.
4. P ERFORMANCE COMPARISON ON COCO DATASET
Several related works in zero-shot detection and open-vocabulary detection are evaluated on COCO.
To compare with them, we train and evaluate ViLD variants following the benchmark setup in Zareian et al.
and report box AP with an IoU threshold of 0.5.
We use the ResNetbackbone, shorten the training schedule to 45,000 iterations, and keep other settings the same as our experiments on LVIS.
Table 4 summarizes the results.
ViLD outperforms Zareian et al.
(2021) by 4.8 Novel AP and 13.5 Base AP.
Different from Zareian et al.
(2021), we do not have a pretraining phase tailored for detection.
Instead, we use an off-the-shelf classiﬁcation model.
The performance of ViLD-text is low because only 48 base categories are available, which makes generalization to novel categories challenging.
In contrast, ViLD-image and ViLD, which can distill image features of novel categories, outperform all existing methods.
4. T RANSFER TO OTHER DATASETS
Trained ViLD models can be transferred to other detection datasets, by simply switching the classiﬁer to the category text embeddings of the new datasets.
For simplicity, we keep the background embedding trained on LVIS.
We evaluate the transferability of ViLD on PASCAL VOC (Everingham et al., 2010), COCO (Lin et al., 2014), and Objects365 (Shao et al., 2019).
Since the three datasets have much smaller vocabularies, category overlap is unavoidable and images can be shared among datasets, e.g ., COCO and LVIS.
As shown in Table 5, ViLD achieves better transfer performance than ViLD-text.
In PASCAL and COCO, the gap is large.
This improvement should be credited to visual distillation, which better aligns region embeddings with the text classiﬁer.
We also compare with supervised training and ﬁnetuning the classiﬁcation layer.
Although across datasets, ViLD has 3-6 AP gaps compared to the ﬁnetuning method and larger gaps compared to the supervised method, it is the ﬁrst time we can directly transfer a trained detector to different datasets using language.
[Table]
L V I S nov e l
L V I S
b a s e + nov e l
T r a n s f e r t o
C O C
O
T r a n s f e r t o
O b j ec t s
4. Q UALITATIVE RESULTS
In Fig.
[Table]
we visualize ViLD’s detection results.
It illustrates ViLD is able to detect objects of both novel and base categories, with high-quality mask predictions for novel objects, e.g ., it well separates banana slices from the crepes (novel category).
We also show qualitative results on COCO and Objects365, and ﬁnd ViLD generalizes well.
We show more qualitative results, e.g ., interactive detection and systematic expansion, in Appendix A.
C ONCLUSION
We present ViLD, an open-vocabulary detection method by distilling knowledge from openvocabulary image classiﬁcation models.
ViLD is the ﬁrst open-vocabulary detection method evaluated on the challenging LVIS dataset.
It attains 16.1 AP r , which surpasses its supervised counterpart at the same inference speed.
With a stronger teacher model (ALIGN), the performance can be further improved to 26.3.
We demonstrate the detector learned from LVIS can be directly transferred to 3 other detection datasets.
Finally, ViLD shows a scalable alternative approach for detecting long-tailed categories instead of collecting expensive detection annotations.
E THICS S TATEMENT
Our paper studies open-vocabulary object detection, a sub-ﬁeld in computer vision.
Our method is based on knowledge distillation, a machine learning technique that has been extensively used in computer vision, natural language processing, etc .
All of our experiments were conducted on public datasets with pretrained models that are either publicly available or introduced in published papers.
The method proposed in our paper is a principled method for open-vocabulary object detection that can be used in a wide range of applications.
Therefore, the ethical impact of our work would primarily depends on the speciﬁc applications.
We foresee positive impacts if our method is applied to object detection problems where the data collection is difﬁcult to scale, such as detecting rare objects for self-driving cars.
But the method can also be applied to other sensitive applications that could raise ethical concerns, such as video surveillance systems
R EPRODUCIBILITY S TATEMENT
We provide detailed descriptions of the proposed method in Sec. 3.
Details about experiment settings, hyper-parameters and implementations are presented in Sec. 4, Appendix C and Appendix D.
We release our code and pretrained models at https://github.com/tensorflow/tpu/ tree/master/models/official/detection/projects/vild to facilitate the reproducibility of our work.
A PPENDIX A A DDITIONAL QUALITATIVE RESULTS
On-the-ﬂy interactive object detection: We tap the potential of ViLD by using arbitrary text to interactively recognize ﬁne-grained categories and attributes.
We extract the region embedding and compute its cosine similarity with a small set of on-the-ﬂy arbitrary texts describing attributes and/or ﬁne-grained categories; we apply a softmax with temperature on top of the similarities.
To our surprise, though never trained on ﬁne-grained dog breeds (Fig. 5), it correctly distinguishes husky from shiba inu.
It also works well on identifying object colors (Fig. 1).
The results demonstrate knowledge distillation from an open-vocabulary image classiﬁcation model helps ViLD to gain understanding of concepts not present in the detection training.
Of course, ViLD does not work all the time, e.g ., it fails to recognize poses of animals.
[Figure]
Systematic expansion of dataset vocabulary: In addition, we propose to systematically expand the dataset vocabulary ( v = { v 1 , ..., v p } ) with a set of attributes ( a = { a 1 , ..., a q } ) as follows:
[Equation]
where e r denotes the region embedding.
We assume v i ⊥⊥ a j | e r , i.e ., given e r the event the object belongs to category v
i is conditionally independent to the event it has attribute a j .
Let τ denote the temperature used for softmax and T denote the text encoder as in Eq.
Then
[Equation]
In this way, we are able to expand p vocabularies into a new set of p × q vocabularies with attributes.
The conditional probability approach is similar to YOLO9000 (Redmon & Farhadi, 2017).
We show a qualitative example of this approach in Fig.
6, where we use a color attribute set as a .
Our open-vocabulary detector successfully detects fruits with color attributes.
We further expand the detection vocabulary to ﬁne-grained bird categories by using all 200 species from CUB-200-2011 (Wah et al., 2011).
Fig.
7 shows successful and failure examples of our openvocabulary ﬁne-grained detection on CUB-200-2011 images.
In general, our model is able to detect visually distinctive species, but fails at other ones.
Transfer to PASCAL VOC:
In Fig. 8, we show qualitative results of transferring an openvocabulary detector trained on LVIS (Gupta et al., 2019) to PASCAL VOC Detection (2007 test set)
(Everingham et al., 2010), without ﬁnetuning (Sec. 4.6 in the main paper).
Results demonstrate that the transferring works well.
Failure cases: In Fig.
9, we show two failure cases of ViLD.
The most common failure cases are the missed detection.
A less common mistake is misclassifying the object category.
We show a failure case of mask prediction on PASCAL VOC in Fig.
It seems that the mask prediction is sometimes based on low-level appearance rather than semantics.
orange: 0. orange: 0.
orange: 0. orange: 0.
orange: 0. orange: 0.
orange: 0.
orange: 0. lemon: 0.
kiwi: 0. kiwi: 0.
lemon: 0. kiwi: 0.
orange: 0.
orange: 0.
(a) Using LVIS vocabulary.
"yellow" lemon: 0.
"green" kiwi: 0.
"yellow" lemon: 0. "dark orange" orange: 0.
"dark orange" orange: 0.
"dark orange" orange: 0.48 "dark orange" orange: 0.
"dark orange" orange: 0.
"dark orange" orange: 0.
"green" kiwi: 0.45 "light orange" orange: 0.
"light orange" orange: 0.
"red orange" orange: 0.
"green" kiwi: 0.
"red orange" orange: 0.
[Figure]
vocabulary
[Figure]
color attributes.
[Figure]
B A NALYSIS OF CLIP ON CROPPED REGIONS
In this section, we analyze some common failure cases of CLIP on cropped regions and discuss possible ways to mitigate these problems.
Visual similarity: This confusion is common for any classiﬁers and detectors, especially on large vocabularies.
In Fig.
11(a), we show two failure examples due to visual similarity.
Since we only use a relatively small ViT-B/32 CLIP model, potentially we can improve the performance with a higher-capacity pretrained model.
In Table 6, when replacing this CLIP model with an EfﬁcientNetl2 ALIGN model, we see an increase on AP.
[Figure]
Aspect ratio: This issue is introduced by the pre-processing of inputs in CLIP.
We use the ViTB/32 CLIP with a ﬁxed input resolution of 224 × 224 .
It resizes the shorter edge of the image to 224, and then uses a center crop.
However, since region proposals can have more extreme aspect ratios than the training images for CLIP, and some proposals are tiny, we directly resize the proposals to that resolution, which might cause some issues.
For example, the thin structure in Fig.
11(b) right will be highly distorted with the pre-processing.
And the oven and fridge can be confusing with the distorted aspect ratio.
There might be some simple remedies for this, e.g ., pasting the cropped region with original aspect ratio on a black background.
We tried this simple approach with both
CLIP and ALIGN.
Preliminary results show that it works well on the fully convolutional ALIGN, while doesn’t work well on the transformer-based CLIP, probably because CLIP is never trained with black image patches.
Multiple objects in a bounding box: Multiple objects in a region interfere CLIP’s classiﬁcation results, see Fig.
11(c), where a corner of an aquarium dominates the prediction.
This is due to
CLIP pretraining, which pairs an entire image with its caption.
The caption is usually about salient objects in the image.
It’s hard to mitigate this issue at the open-vocabulary classiﬁcation model’s end.
On the other hand, a supervised detector are trained to recognize the object tightly surrounded by the bounding box.
So when distilling knowledge from an open-vocabulary image classiﬁcation model, keeping training a supervised detector on base categories could help, as can be seen from the improvement of ViLD over ViLD-image (Sec. 4.4).
Conﬁdence scores predicted by CLIP do not reﬂect the localization quality: For example, in
Fig.
12(a), CLIP correctly classiﬁes the object, but gives highest scores to partial detection boxes.
CLIP is not trained to measure the quality of bounding boxes.
Nonetheless, in object detection, it is important for the higher-quality boxes to have higher scores.
In Fig. 12(c), we simply re-score by taking the geometric mean of CLIP conﬁdence score and the objectness score from the proposal model, which yields much better top predictions.
In Fig.
12(b), we show top predictions of the Mask
[Table]
R-CNN model.
Its top predictions have good bounding boxes, while the predicted categories are wrong.
This experiment shows that it’s important to have both an open-vocabulary classiﬁcation model for better recognition, as well as supervision from detection dataset for better localization
[Table]
C A DDITIONAL QUANTITATIVE RESULTS
Hyperparameter sweep for visual distillation:
Table 7 shows the parameter sweep of different distillation weights using L 1 and L 2 losses.
Compared with no distillation, additionally learning from image embeddings generally yields better performance on novel categories.
We ﬁnd L 1 loss can better improve the AP r performance with the trade-off to AP c and AP f .
This suggests there is a competition between ViLD-text and ViLD-image.
[Table]
Box APs and ResNet-152 backbone: Table 8 shows the corresponding box AP of Table 3 in the main paper.
In general, box AP is slightly higher than mask AP.
In addition, we include the results of ViLD variants with ResNet-152 backbone.
The deeper backbone improves all metrics.
The trend/relative performance is consistent for box and mask APs, as well as for different backbones.
ViLD-ensemble achieves the best box and mask AP r .
Ablation study on prompt engineering: We conduct an ablation study on prompt engineering.
We compare the text embeddings ensembled over synonyms and 63 prompt templates (listed in
Appendix D) with a non-ensembled version: Using the single prompt template “a photo of { article }
[Figure] [Figure]
{ category } ”.
Table 9 illustrates that ensembling multiple prompts slightly improves the performance by 0.4 AP r .
[Table] [Table]
D M ORE IMPLEMENTATION DETAILS
ViLD-ensemble architecture:
In Fig. [Table] we show the detailed architecture and learning objectives for ViLD-ensemble.
The ensembling technique is introduced in Sec. 3.4.
Model used for qualitative results: For all qualitative results, we use a ViLD model with ResNet152 backbone, whose performance is shown in Table 8.
Details for supervised baselines: For a fair comparison [Table] we train the second stage box/mask prediction heads of Supervised and Supervised-RFS baselines in the class-agnostic manner introduced in Sec. 3.1.
Details for R-CNN style experiments: We provide more details here for the R-CNN style experiments, CLIP on cropped regions in Sec.
4.2 and ViLD-text+CLIP in Sec.
Generalized object proposal: We use the standard Mask R-CNN R50-FPN model.
To report mask AP and compare with other methods, we treat the second-stage reﬁned boxes as proposals and use the corresponding masks.
We apply a class-agnostic NMS with 0.9 threshold, and output a maximum of 1000 proposals.
The objectness score is 1 - the background score.
2) Open-vocabulary classiﬁcation on cropped regions: After obtaining CLIP conﬁdence scores for the 1000 proposals, we apply a class-speciﬁc
NMS with a threshold of 0.6, and output the top 300 detections as the ﬁnal results.
Additional details for ViLD variants: Different from the R-CNN style experiments, for all ViLD variants (Sec. 3.3, Sec. 3.4), we use the standard two-stage Mask R-CNN with the class-agnostic localization modules introduced in Sec. 3.1.
Both the M ofﬂine proposals and N online proposals are obtained from the ﬁrst-stage RPN (Ren et al., 2015).
In general, the R-CNN style methods and
ViLD variants share the same concept of class-agnostic object proposals.
We use the second-stage outputs in R-CNN style experiments only because we want to obtain the Mask AP, the main metric, to compare with other methods.
For ViLD variants, we remove the unnecessary complexities and show that using a simple one-stage RPN works well.
[Figure]
Architecture for open-vocabulary image classiﬁcation models: Popular open-vocabulary image classiﬁcation models (Radford et al., 2021; Jia et al., 2021) perform contrastive pre-training on a large number of image-text pairs.
Given a batch of paired images and texts, the model learns to maximize the cosine similarity between the embeddings of the corresponding image and text pairs while minimizing the cosine similarity between other pairs.
Speciﬁcally, for CLIP (Radford et al., 2021), we use the version where the image encoder adopts the Vision Transformer (Dosovitskiy et al., 2020) ViT-B/32 architecture and the text encoder is a Transformer (Vaswani et al., 2017).
For
ALIGN (Jia et al., 2021), its image encoder is an EfﬁcientNet (Tan & Le, 2019) and its text encoder is a BERT (Devlin et al., 2019).
Details for ViLD with stronger teacher models:
In both experiments with CLIP ViT-L/14 and
ALIGN, we use EfﬁcientNet-b7 as the backbone and ViLD-ensemble for better performance.
We also crop the RoI features from only FPN level P 3 in the feature pyramid.
The large-scale jittering range is reduced to [0.5, 2.0].
For CLIP ViT-L/14, since its image/text embeddings have 768 dimensions, we increase the FC dimension of the Faster R-CNN heads to 1,024, and the FPN dimension to 512.
For ViLD w/ ALIGN, we use the ALIGN model with an EfﬁcientNet-l2 image encoder and a BERT-large text encoder as the teacher model.
We modify several places in the Mask R-CNN architecture to better distill the knowledge from the teacher.
We equip the ViLD-image head in
ViLD-ensemble with the MBConvBlocks in EfﬁcientNet-l2.
Since the MBConvBlocks are fullyconvolutional, we apply a global average pooling to obtain the image embeddings, following the teacher.
The ViLD-text head keeps the same Faster R-CNN head architecture as in Mask R-CNN.
Since ALIGN image/text embeddings have 1,376 dimensions (2.7 × CLIP embedding dimension), we increase the number of units in the fully connected layers of the ViLD-text head to 2,048, and the FPN dimension to 1,024.
Text prompts: Since the open-vocabulary classiﬁcation model is trained on full sentences, we feed the category names into a prompt template ﬁrst, and ensemble various prompts.
Following Radford et al.
(2021), we curate a list of 63 prompt templates.
We specially include several prompts containing the phrase “in the scene” to better suit object detection, e.g ., “There is { article } { category } in the scene”.
Our list of prompt templates is shown below: ’There
is {article} {category} in the scene.
There is the {category} in the scene.
’a photo of {article} {category} in the scene. ’a photo of the {category} in the scene.
’a photo of one {category} in the scene.
’itap of {article} {category}.
’itap of my {category}.’ ’
itap of the {category}.’
a photo of {article} {category}.’ ’ a photo of my {category}.’ ’
a photo of the {category}.’ ’ a photo of one {category}.’ ’
a photo of many {category}.’ ’
a good photo of {article} {category}.’ ’ a good photo of the {category}.’ ’
a bad photo of {article} {category}.’ ’ a bad photo of the {category}.’ ’
a photo of a nice {category}.’ ’ a photo of the nice {category}.’ ’
a photo of a cool {category}.’ ’ a photo of the cool {category}.’ ’
a photo of a weird {category}.’ ’ a photo of the weird {category}.’ ’
a photo of a small {category}.’ ’ a photo of the small {category}.’ ’
a photo of a large {category}.’ ’ a photo of the large {category}.’ ’
a photo of a clean {category}.’ ’ a photo of the clean {category}.’ ’
a photo of a dirty {category}.’ ’ a photo of the dirty {category}.’ ’
a bright photo of {article} {category}.’ ’ a bright photo of the {category}.’ ’
a dark photo of {article} {category}.’ ’ a dark photo of the {category}.’ ’
a photo of a hard to see {category}.’ ’ a photo of the hard to see {category}.’ ’
a low resolution photo of {article} {category}.’ ’ a low resolution photo of the {category}.’ ’
a cropped photo of {article} {category}.’ ’ a cropped photo of the {category}.’ ’
a close-up photo of {article} {category}.’ ’ a close-up photo of the {category}.’ ’
a jpeg corrupted photo of {article} {category}.’ ’ a jpeg corrupted photo of the {category}.’ ’
a blurry photo of {article} {category}.’ ’ a blurry photo of the {category}.’ ’
a pixelated photo of {article} {category}.’ ’ a pixelated photo of the {category}.’ ’
a black and white photo of the {category}.’ ’ a black and white photo of {article} {category}.’ ’
a plastic {category}.’ ’ the plastic {category}.’ ’
a toy {category}.’ ’ the toy {category}.’ ’
a plushie {category}.’ ’ the plushie {category}.’ ’
a cartoon {category}.’ ’ the cartoon {category}.’ ’
an embroidered {category}.’ ’ the embroidered {category}.’ ’
a painting of the {category}.’ ’ a painting of a {category}.’