I MAGINARY N ET : L EARNING O BJECT D ETECTORS WITHOUT R EAL I MAGES AND A NNOTATIONS
Anonymous authors
Paper under double-blind review
A BSTRACT
Without the demand of training in reality, humans are able of detecting a new category of object simply based on the language description on its visual characteristics.
Empowering deep learning with this ability undoubtedly enables the neural network to handle complex vision tasks, e.g. , object detection, without collecting and annotating real images.
To this end, this paper introduces a novel challenging learning paradigm Imaginary-Supervised Object Detection (ISOD), where neither real images nor manual annotations are allowed for training object detectors.
To resolve this challenge, we propose I MAGINARY N ET , a framework to synthesize images by combining pretrained language model and text-to-image synthesis model.
Given a class label, the language model is used to generate a full description of a scene with a target object, and the text-to-image model is deployed to generate a photo-realistic image.
With the synthesized images and class labels, weakly supervised object detection can then be leveraged to accomplish ISOD.
By gradually introducing real images and manual annotations, I MAGINARY N ET can collaborate with other supervision settings to further boost detection performance.
Experiments show that I MAGINARY N ET can (i) obtain about 70% performance in ISOD compared with the weakly supervised counterpart of the same backbone trained on real data, (ii) significantly improve the baseline while achieving state-of-the-art or comparable performance by incorporating I MAGINARY N ET with other supervision settings. Our code will be publicly available.
I NTRODUCTION
Without the demand of training in reality, humans are able of detecting a new category of object simply based on the language description on its visual characteristics.
Equipping this ability to deep learning may allow the neural network to handle complex vision tasks, e.g. , object detection, without real images and annotations.
Recently, we witness the rise of Contrastive Language-Image
Pre-training (CLIP) (Radford et al., 2021), where general knowledge can be learned by pre-training and then be applied to various downstream tasks via zero-shot learning or task-specific fine-tuning.
Unlike image classification, object detection is more challenging and has a larger gap than pretraining tasks.
Several methods, such as RegionCLIP (Zhong et al., 2022a), ViLD (Gu et al., 2021), and Detic (Zhou et al., 2022), have been suggested to transfer knowledge from pre-trained CLIP (Radford et al., 2021) to some modules of detection.
However, real images and annotations are still required for some key modules of the object detectors, such as Region Proposal Network (RPN) or
Region of Interest (RoI) heads.
In this work, we aim to raise and answer a question: given suitable pre-trained models, can we learn object detectors without real images and manual annotations?
To this end, we introduce a novel learning paradigm, i.e. , Imaginary-Supervised Object Detection (ISOD), where no real images and manual annotations can be used for training object detection.
Fortunately, benefited from the progress in vision-language pre-training, ISOD is practically feasible.
Here we propose I MAGINARY N ET , a framework to learn object detectors by combining pretrained language model as well as text-to-image synthesis models.
In particular, the text-to-image synthesis model is adopted to generate photo-realistic images, and the language model can be used to improve the diversity and provide class labels for the synthesized images.
Then, ISOD can be conducted by leveraging weakly supervised object detection (WSOD) algorithms on the synthesized images with class labels to learn object detectors.
We set up a strong CLIP-based model as the baseline to verify the effectiveness of I MAGINARY N ET .
Experiments show that I MAGINARY N ET can outperform the CLIP-based model with a large margin.
Moreover, I MAGINARY N ET obtains about 70% performance in ISOD compared with the weakly supervised model of the same backbone trained on real data, clearly showing the feasibility of learning object detection without any real images and manual annotations.
By gradually introducing real images and manual annotations, I MAGINARY N ET can collaborate with other supervision settings to further boost detection performance.
It is worthy noting that the performance of existing object detection models may be constrained by the limited amount of training data.
As a result, we use I MAGINARY N ET as a dataset expansion approach to incorporate with real images and manual annotations.
Further experiments show that I MAGINARY N ET significantly improves the performance of the baselines while achieving state-of-the-art or comparable performance in the supervision setting.
To sum up, the contributions of this work are as follows:
• We propose I MAGINARY N ET , a framework to generate synthesized images as well as supervision information for training object detector.
To the best of our knowledge we are among the first work to train deep object detectors solely based on synthesized images.
• We propose a novel paradigm of object detection, Imaginary-Supervised Object Detection (ISOD), where no real images and annotations can be used for training object detectors.
We set up the benchmark of ISOD and obtain about 70% performance in ISOD when compared with the WSOD model of the same backbone trained on real data.
• By incorporating with real images and manual annotations, ImaginaryNet significantly improves the baseline model while achieving state-of-the-art or comparable performance.
R ELATED W ORK 2. O BJECT D ETECTION
Most fully-supervised object detection methods (FSOD) (Ren et al., 2015;
Redmon et al., 2016; Tian et al., 2019; Carion et al., 2020) rely on large amount of training data with box-level annotations.
To reduce the annotation costs, some works attempt to train a detector with incompletely supervised training data.
For example, weakly-supervised object detection (WSOD) (Huang et al., 2022; Dong et al., 2021; Bilen & Vedaldi, 2016;
Tang et al., 2017) requires only image-level labels.
While semisupervised object detection (SSOD) (Liu et al., 2021;
Xu et al., 2021; Chen et al., 2022) leverages unlabeled data combining with box-level labeled data.
Although these works use relatively less or weak supervision, all of these works still rely on real images and manual annotations.
In this paper, we propose ISOD, where no real images and manual annotations can be used for training object detection, thereby saving the demand for data acquisition and annotation costs.
2. S IM 2R EAL
Sim2real aims to use the simulator engines to simulate images for the model.
Some works (Akhyani et al., 2022; Sadeghi & Levine, 2016; Wang et al., 2018) use engines to simulate data for different vision tasks, such as emotion classification. In object detection, Horv´ath et al.
; Borrego et al.
; Tremblay et al.
(2018) use simulated images with 2D or 3D engine, e.g., UE4, for industrial object detection.
Because of the large domain gap caused by limitation of simulating engine, Sim2real and subsequent domain adaption methods focus on reducing domain gap. However, with the progress in text-to-image synthesis, domain gap of images generated by such models has been largely reduced.
The key problem is how to effectively employ text-to-image synthesis model for generating diverse images with proper content and quality. To this end, we propose I MAGINARY N ET , which can even archive 70% performance in ISOD compared with the weakly supervised counterpart of the same backbone.
2. P RE - TRAINED M ODELS
Pre-trained models have shown effectiveness in many vision tasks including object detection.
Some
CLIP-based models can transfer knowledge from pre-trained CLIP (Radford et al., 2021) model.
However, real images and manual annotations are still needed for their RPNs or RoI heads.
Visual synthesis models, such as GAN (Goodfellow et al., 2016), StyleGAN (Karras et al., 2019), and
[Figure]
Language M ode l wavingwings besidethebeach standing with others
A photo of a bird standing with others.
ℒ !"
# ℒ $%&
ImaginaryNet
R ep r e s en t a t i on
G ene r a t
o r
T e x t - t o - I m age
S y n t he s i s
M ode l
D e t e c t i on H ead
LanguageModel
[Figure]
RepresentationGenerator
[Figure]
ImageBART Esser et al.
(2021) aim to generate plausible images.
In recent years, we witness the rise of text-to-image synthesis models with high quality and capability of language controlling, such as DALL-E (Ramesh et al., 2021), Stable Diffusion (Rombach et al., 2022), and Imagen (Saharia et al., 2022).
However, few studies have been given to only using the knowledge from pre-trained text-to-image synthesis models to handle complex vision tasks, e.g. , object detection.
M ETHODOLOGY 3. P RELIMINARIES AND P ROBLEM F ORMULATIONS
Object detection aims to find each object with the bounding box b k ∈ R 4 and class label c k ∈ C in a given image I ∈ R C × W × H , where C , W and H denote the channels, width and height of the image and k denotes the k -th object, respectively.
In standard training process of object detection, D R is the dataset which contains real images and their corresponding annotations.
In practice, D R may contain different types of data and annotations, where D R = D R s ∪ D R w ∪ D R u .
Class-labeled real data D R w =
{ ( I , { ( c ) i } ) k } contains real images and corresponding class labels, which is used for WSOD setting.
Box-labeled real data D R s = { ( I , { ( b , c ) i } ) k } contains real images, bounding box and class label of each object.
Un-annotated real data D R u = { ( I ) k } only contains real images.
Box-level annotated and un-annotated real data are used in SSOD setting.
In this paper, we propose Imaginary-Supervised Object Detection (ISOD), where no real images and annotations can be accessed, i.e. , D R = ∅ .
3. I MAGINARY G ENERATOR
I MAGINARY N ET aims to generate synthesized images as well as supervision information for training object detectors.
Given a class vocab C , I MAGINARY N ET samples uniformly to get a class label c ∈ C .
Then, the class name of c will be fulfilled into a prefix phrase, which is “ A photo of a ”, to obtain the subject description P c of the imaginary image.
To make the scene of image more specific and diverse, P c will be the guidance prefix of the language model G L ,
e.g. , GPT-2 (Radford et al., 2019), to imagine the full description of the scene of image.
The language model G L , which is a transformer-based autoregressive network, generates the full description token by token until meeting a full stop or exceeding the maximum of length M L 1 .
Let t c =
G L ( P c ) be the generated full description of the imaginary image.
I MAGINARY N ET further obtains the synthesized image by
[Equation] [Figure]
MIL Branch ! " %" " " % " " !
PGT '!
Refine Branch " !"
MIL Branch ! " %" " " % " " ! PGT '! Refine Branch ! !" GT
Teacher
[Figure]
PGT
[Figure]
Student
[Figure]
Real
[Figure]
Imaginary
[Figure]
Real * ( * ) Imaginary EMAUpdate where z ∼ N (0 , 1) is random noise sampled from normal distribution. The text-to-image synthesis model G V usually adopts a transformer-based autoregressive model or diffusion network, and generates imaginary image I based on z with the guidance of t c in a step by step manner.
For the imaginary image I , I MAGINARY N ET obtains its feature map h
= E I ( I ) , where E I is the image encoder consisting of multi-layers of convolutional networks.
Simultaneously, proposals
P =
R ( I ) of the imaginary image I are extracted by proposal generator R , which adopts a selective search module or an RPN without initilization.
I MAGINARY N ET applies RoI Pooling (Ren et al., 2015; Girshick, 2015) to each proposal p i ∈ P in feature map h to obtain the proposal representation f i ∈ R d .
Consequently, I MAGINARY N ET obtains a set of imaginary proposal representations { f i } with their corresponding class-label c , which allows ISOD to learn object detectors without real images and annotations.
3. D ETECTION N ETWORK
We add an object detection head at the end of the Representation Generator , which takes the representation f i as input and predicts the bounding box of an object.
In standard object detection, training images and corresponding annotations are given, formulating a fully-supervised training paradigm (FSOD).
Instead, we propose a new learning paradigm named ISOD, in which neither real training images nor manual annotations can be reached.
This means that we cannot apply the traditional approach (Ren et al., 2015; Tian et al., 2019) straightforwardly.
Fortunately, our
I MAGINARY N ET samples a target class c describing the semantics of the corresponding imaginary image.
Note that both the set of proposal representation { f i } and image-level class label c are available.
Thus, ISOD can be conducted by leveraging weakly-supervised object detection (WSOD) algorithms on the proposal representations and class labels obtained from I MAGINARY N ET .
Following OICR (Tang et al., 2017), the object detection head consists of a Multiple Instances
Learning (MIL) branch and a Refinement branch with the structure shown in Fig (2).
During the training phase, all object representations are first fed into the MIL branch.
The MIL branch combines two parallel classification streams, each of which contains a softmax function.
One softmax function is along the classes dimension to output the score matrix x mc ∈ R |P|×| C | , and the other is along the object dimension to output the score matrix x mr ∈ R |P|×| C | .
Subsequently, we compute the element-wise product of these two scores matrix and get the x m = x
mc ⊙ x mr . x m indicates the final object-level classification scores.
Then we get the c -th class image-level score s mc by sum over all object: s mc = (cid:80) |P| i =
1 x mi,c .
We convert the target label c into one-hot form ˆ y and calculate the Binary
Cross Entropy (BCE) loss:
[Equation]
To further improve the performance of the detector, we integrate the self-training methodology and construct the Refinement branch, which has a similar structure as the RoI head of the R-CNN-like detector (Ren et al., 2015).
Following Tang et al. , we keep the proposal with the highest classification score from x m as the pseudo-ground-truth (PGT), which are used to optimize the
Refinement branch by a refinement loss L ignref .
Finally, the learning objective of our object detection head of ISOD is:
[Equation]
We consider two baseline methods, a strong baseline based on unsupervised CLIP model, and an upper-bound baseline based on OICR trained on 5,000 real images with image-level annotations.
M ETHODS [Table] The whole training pipeline of ISOD is shown in Fig. 2(a).
Here we take the OICR as the example and it is feasible to employ other WSOD methods for the end of ISOD.
Furthermore, by gradually introducing real images and manual annotations, I MAGINARY N ET can collaborate with other supervision setting to boost detection performance.
Here we present two examples by incorporating I MAGINARY N ET with weakly supervision and full supervision on real data.
During training, both real images and synthesized images are fed into the Representation
Generator and object detection head in order and optimize these two modules via back-propagation.
And we only need to slightly modify the training pipeline of object detection head.
When real images and image-level annotations are available, i.e. , D R w ̸ = ∅ , as shown in Fig.
2(b) we can uniformly sample synthesized and real images and feed them into the WSOD training process.
When both real images and box-level annotations are available, i.e. , D R s ̸ = ∅ , we remove the MIL branch and treat the synthesis images as unlabeled image .
As shown in Fig. 2(c), we simply leverage the teacherstudent framework widely adopted in SSOD to train the detection head.
The teacher detection network generates pseudo-ground-truth (PGT) for unlabeled real images and imaginary images to train the student detection network.
In addition, the student is also trained on real images with box-level ground-truth (GT) by a supervised-learning manner (Ren et al., 2015).
3. T RAINING AND I NFERENCE P IPELINE
During the training phase, I MAGINARY N ET samples a target class c and generates the full description t via a language model.
Then we get an imaginary image by Eq.
(1) as well as its feature map h by h =
E I ( I ) .
Consequently, a set of region proposals is generated by the Proposals Generator, and proposal representations are obtained via RoI Pooling.
Finally, the proposal representations are fed into different object detection heads according to the supervision settings.
During the whole training process, only the parameters of the Representation Generator and Object Detection head need to be updated.
In the inference phase, the language model and synthesis model will be abandoned.
Taking a real image as the input, the Representation Generator generates proposal representations, and the detection head decodes them to predict the detection bounding box.
E XPERIMENTS 4. I MPLEMENTATION D ETAILS
We use GPT-2 (Radford et al., 2019) as the language model and DALLE-mini, which can better follow the language guidance, as the text-to-image synthesis model.
We implement the image encoder with ResNet50 pretrained on ImageNet dataset.
For Proposal Generator, we use Selective
Search following W2N
(Huang et al., 2022) for obtaining no real images and manual annotations.
All training hyper-parameters follow OICR (Tang et al., 2017) and W2N (Huang et al., 2022) for a fair comparison.
Unless otherwise specified, we generate 5,000 imaginary images during training, which has the similar amount of images in comparison to PASCAL VOC2007 trainval set.
4. E XPERIMENTAL S ETUP
We first compare I MAGINARY N ET with the ISOD model to verify whether it is feasible to learn object detectors without real images and manual annotations.
To the best of our knowledge, we are among the first to investigate ISOD solely based on synthesized images.
So we set up a strong baseline based on the CLIP model.
For CLIP baseline, we use the Edge Boxes (Zitnick & Doll´ar, 2014) algorithm 2 to extract potential proposals on VOC test images for CLIP to classify its class, [Table] based on its similarity score with pre-designed prompts.
Before computing AP, we apply the NonMaximum Suppression (NMS) operation to remove redundant proposals based on CLIP scores.
See
Appendix A.1 for more details.
We also assess I MAGINARY N ET by collaborating with WSOD method W2N (Huang et al., 2022), i.e. , I MAGINARY N ET (WSOD).
For comparison, we consider WSDDN (Bilen & Vedaldi, 2016),
OICR (Tang et al., 2017) and W2N (Huang et al., 2022) on real images as the baseline detection networks.
Compared with them, I MAGINARY N ET in ISOD, i.e. , I MAGINARY N ET (ISOD), adopts similar training pipeline but does not require real images and manual annotations, whereas I MAGINARY N ET (WSOD) leverages both real and synthesized images.
4. O VERALL R ESULTS 4.3. I MAGINARY - SUPERVISED O BJECT D ETECTION
As shown in Tab. 1, I MAGINARY N ET (ISOD) achieves an mAP of 33.23.
This verifies the feasibility of learning object detectors without real images and annotations.
Moreover, I MAGINARY N ET (ISOD) outperforms the CLIP baseline model with a gain of 12.39 in mAP.
This further supports that the competitiveness of a pre-trained generative model against pre-trained contrastive alignment models ( e.g. , CLIP) for learning object detectors.
For most classes, I MAGINARY N ET (ISOD) can obtain a significant performance gain, and I MAGINARY N ET (ISOD) has much fewer parameter amounts and computational cost in inference because the pre-trained modules can be abandoned after training.
Especially, for some classes such as cat, the performance improved from 25.92 of CLIP to 66.72 of I MAGINARY N ET (ISOD).
The results clearly show the feasibility of I MAGINARY N ET (ISOD), i.e. , learning object detectors without real images and manual annotations.
We further compare I MAGINARY N ET (ISOD) with an upper-bound baseline, WSOD on real images with class-level annotations.
For a fair comparison, we only generate 5,000 imaginary samples during training.
Compared with OICR, which has the same backbone with Ours and trained on 5, real images and manual annotations in WSOD setting, we achieved about 70% performance of it.
Considering that synthesized images and image-level annotations are free for I MAGINARY N ET , our method has the potential to further improve detection accuracy by increasing training samples.
MSCOCO Results We also conduct the experiments on MSCOCO dataset, and obtain similar observations as VOC2007.
See Appendix A.2 for more details.
4.3. I NCORPORATING WITH WSOD ON R EAL I MAGES AND MANUAL A NNOTATIONS
We further show that the synthesized images by I MAGINARY N ET can collaborate with other supervision settings on real images and manual annotations for boosting detection performance.
Here we use WSOD as an example.
As given in Tab. 2, I MAGINARY N ET (WSOD) has achieved state-ofthe-art performance in WSOD.
This shows the effectiveness of I MAGINARY N ET can not only learn object detectors without real images and annotations, but also is effective in improving the performance of backbones without increasing parameters and inference cost.
I MAGINARY N ET (WSOD) obtains an mAP of 64.45, which is higher than the state-of-the-art WSOD models.
4. C OMPARISON WITH R EAL D ATA
How effective is I MAGINARY N ET in comparison with real data?
To answer this question, we conduct comparison experiments in SSOD setting.
We use the same backbone, R ES N ET 50-FPN as the can play a similar role in enhancing performance.
10K Imaginary data can improve the model by a similar margin compared with 10K real data.
Moreover, introducing imaginary data when extra real data exists can further improve the performance.
[Table]
We can observe that features can cluster based on the object type.
This shows imaginary data contains similar knowledge like real data.
[Table] [Figure]
supervised model F ASTER RCNN as well as the SSOD model U NBIASED T EACHER 3 .
For fairness, we use the same backbone, hyper-parameters, and real data by following the competing methods.
As shown in Tab.
3, comparing with F ASTER -RCNN and U NBIASED T EACHER , I MAGINARY N ET (SSOD) has achieved comparable performance, about 99.3% performance of U NBIASED
T EACHER .
This shows that generated data are almost as effective as real data but have much lower cost in data acquisition and higher controllability in class balance.
Compared to the baseline U NBIASED T EACHER , we can observe that generated data are orthogonal with real data, where we can improve the performance by increasing both real and synthesized data.
4. V ISUALIZATION
To figure out why I MAGINARY N ET can learn object detectors without real images and manual annotations, we conduct several visualizations to explain the reason from different aspects.
4.5. E XTRACTED F EATURES
To verify whether the features extracted from imaginary images contain similar information to real images, we extracted their features from the same Representation Generator.
For real images, we extract 10 features per class as anchors and 10 features per class from imaginary images.
We use T-SNE to project them to a 2D image.
In Fig.
3, we can observe that features of one class from imaginary images are coincident with the feature cluster of the same class from real images.
This indicates that features from imaginary images contain similar information in comparison with real images, which explains why I MAGINARY N ET can learn object detectors even without real images and annotations.
4.5. I MAGINARY I MAGES
We randomly sample imaginary images from the synthesis model and corresponding text descriptions from the language model to observe how they can play the role of real data.
In Fig.
5, we can find that text descriptions are quite diverse.
Meanwhile, high-quality images from the text-to-image synthesis model follow the description well and contain vivid objects of the corresponding class.
Although all images generated are not in real datasets, I MAGINARY N ET successfully simulates the [Figure] [Figure]
[Table]
[Figure] [Table] [Table]
scene of the images from object detection datasets.
This may explain why I MAGINARY N ET can even enhance object detectors under settings that contain real images and annotations.
4.5. C ASE S TUDIES
To better reveal the quality of I MAGINARY N ET , we visualize detection results from VOC2007 test split in ISOD setting.
In Fig.
6, many objects are detected correctly.
In some hard cases, e.g. , the 5-th image which contains many drivers and motors overlapping together, can also be detected well with nearly-correct boxes and classes.
This proves the effectiveness of I MAGINARY N ET in ISOD.
Although I MAGINARY N ET shows pretty effectiveness in many cases, we still find out some failure cases.
For example, in Fig. 6, some motorbikes are not detected well and persons are classified into the wrong class.
This is because ISOD is a very strict setting, where the model cannot access any real images and annotations.
Nonetheless, I MAGINARY N ET still shows that it is feasible to learn object detectors in this hard setting.
A BLATION S TUDIES
To further analyze the performance of I MAGINARY N ET , we conduct several ablation studies to investigate the proposed framework.
5. I NFLUENCE OF T OTAL I MAGINARY S AMPLES
Can the model obtain higher performance if I MAGINARY N ET imagine more samples?
To figure out this, we conduct experiments on ISOD with different scales of imaginary samples.
In Tab. 4 and
Fig. 4, one can see that the overall performance, mAP, increases steadily to 36.28 finally.
The result shows that more imaginary samples can make the detector stronger.
Meanwhile, I MAGINARY N ET obtains mAP of 29.33 with 2K samples only.
This shows the effectiveness of imaginary samples, which can train the detector with very limited samples.
We also notice the fluctuation of several [Figure] [Table] [Figure]
[Table] classes, which may be explained by the instability of class-level annotation learning.
In general, more samples will improve the final performance.
5. E FFECTIVENESS OF L ANGUAGE M ODEL
Whether the various scene descriptions are useful for detector learning?
To figure out this question [Table] we re-train an I MAGINARY N ET without the language model.
In Tab. [Table] we can observe a significant drop in mAP.
The performance of each class also drops in most cases.
Such class, like cat, drops over 30 of AP50.
The result indicates that language model is also important in generating diverse images.
We also notice the unexpected drop in some classes and we believe this can be explained by the failure of the language model to generate a scene similar to the real dataset in some cases.
We will leave this as future work to generate images that are more suitable for learning detectors.
5. V ERIFICATION OF O RTHOGONALITY WITH D ATA A UGMENTATIONS
Whether the data augmentation strategies still work on imaginary images?
To answer the question, we conduct several ablation studies with different data augmentation strategies, training augmentation, and testing augmentation.
For most WSOD method, which are also employed into our ISOD setting, no training augmentation is needed because it is relatively hard to learn with class-labeled images.
Consequently, we conduct this experiment in SSOD setting.
As Tab. 6 shows, we can observe that I MAGINARY N ET (SSOD) obtains better performance after activating training augmentation and testing augmentation.
These results give strong evidence that our framework is orthogonal with data augmentations.
Moreover, orthogonality with data augmentation strategies may imply that imaginary samples have a similar property to real data.
B ROADER I MPACT
I MAGINARY N ET is an effective framework to learn detectors without real images and annotations for further improving the performance with real data.
It can provide new insight for handling complex vision tasks under low-resource regime.
For example, it is possible to learn detection or segmentation models without any real data.
However, the performance of I MAGINARY N ET (ISOD) is still inferior to WSOD and FSOD. Thus, more studies are required to generate diverse and photorealistic images and to better extract supervision information from the language descriptions and synthesized images.
C ONCLUSION
In this paper, we proposed a novel learning paradigm of object detection, Imaginary-Supervised Object Detection (ISOD), where no real images and manual annotations can be used for training object detectors. In particular, we presented an I MAGINARY N ET framework to generate synthesized images as well as supervision information.
Experiments shows that I MAGINARY N ET (ISOD) can obtain about 70% of performance in comparison with the WSOD counterpart trained using real data and image-level annotations.
Moreover, I MAGINARY N ET can collaborate with other supervision settings on real data to further boost detection performance.
A A PPENDIX A. CLIP B ASELINE
We use ViT-B/16 as the image encoder for CLIP and use Edge Boxes to extract 1000 proposals for each test image.
Each proposal will be transformed to 384 × 384 to fit the CLIP input size.
Then each proposal will be calculated similarity with the prompt which consists of the prefix “a photo of” and the class name.
We select the highest CLIP similarity as the class of the proposal.
Before computing AP, we apply the Non-Maximum Suppression (NMS) operation to remove redundant proposals according to their CLIP scores.
A. I MAGINARY - SUPERVISED O BJECT D ETECTION R ESULTS ON MSCOCO
[Table]
We also verified our framework on MSCOCO2014, which is also a common dataset in object detection.
As the results in Tab.
we outperform the CLIP baseline with nearly two times of mAP.
This proves that our framework is also effective on the harder dataset, which contains 80 classes.
A. C OMPARISON OF D IFFERENT V ISUAL S YNTHESIS M ODEL
[Table]
We notice that the generated images are different from different generative models.
In this paper, we use two effective open-source visual synthesis models, S TABLE D IFFUSION and D ALLE - MINI .
We conduct a comparison on ISOD to verify the influence of synthesis models.
In Tab. 8, we can find that D ALLE - MINI can provide higher performance for I MAGINARY N ET .
We explain this as
D ALLE - MINI ’s results contain fewer objects, which are easier for the model to learn in the training [Table] [Table]
A. C OMPARISON WITH O THER CLIP- BASED D ETECTION A PPROACHES
We add the extra comparison with these CLIP-based Detection approaches.
Due to these models, e.g., RegionCLIP (Zhong et al., 2022b), Detic (Zhou et al., 2022)
[Table]
and ViLD (Gu et al., 2021), use different human-annotated real detection data, we adopt the labeled dataset for training each model.
For a fair comparison, we adopt I MAGINARY N ET (SSOD) for leveraging both imaginary data and the same labeled data.
We compare results on 20 base class in VOC 2007 test split.
For ViLD, we directly provide the results from their paper.
model.
The model can create a new object with the description, “an elephant-like creature with wings”.
From the Tab. 9, we can observe I MAGINARY N ET (SSOD) still obtain the best performance.
This proves the effectiveness of generated data from I MAGINARY N ET .
A. D ISCUSSION OF S YNTHESIS M ODEL
Different from class-to-image synthesis models, text-to-image synthesis models have a much stronger capability of composition and extrapolation (Ramesh et al., 2021; Rombach et al., 2022;
Saharia et al., 2022).
For example, the model may be trained with only “black cats” and “white swans”, but it can generate “black swans”.
Meanwhile, text-to-image synthesis models can even create new objects with detailed descriptions.
For example, they can generate the image with the description “an elephant-like creature with wings” (see Fig. 7).
A. D ISCUSSION OF L IMITATIONS
We further discuss the limitations and conduct extra ablation studies on the quantity and quality of the generated images from three aspects: (1) sample number, (2) diversity and photo-reality, and (3) generation errors
. A.6. S AMPLE N UMBER
As the Tab. 4, one can see that I MAGINARY N ET obtains the mAP of 36.28 when using 25K generated samples.
Nonetheless, the growth of performance has slowed down when the number of generated samples is more than 10K.
Thus, we speculate that 36.28 approaches the upper bound of the performance.
We note that it may be hard for our method to surpass the method training on full annotated real images.
How to utilize the generative models effectively for refreshing this upper bound will be a question in the future study.
A.6. D IVERSITY AND P HOTO - REALITY
The diversity of images generated will help the model learn more knowledge and improve its robustness.
Diversity is affected by not only the text-to-image synthesis model but also language prompts.
This is also the motivation that we introduce the language model to the framework to obtain better diversity.
Meanwhile, higher photo-reality also helps the model to obtain a better performance due to low domain gap to real images.
With the incredible increase in text-to-image synthesis models, images that are very similar to real photos can be synthesized.
Therefore, we believe that we can obtain images generated with higher and higher photo-reality.
[Table]
[Table]
A.6. G ENERATION E RRORS
The language and text-to-image synthesis model may bring in errors during their generation process.
To explore errors from language and text-to-image generation, we set up a CLIP-based filter, which measures the quality of generated data.
Following the method of CLIP classification (Radford et al., 2021), the model will find out the most matched text in the group { “A photo of aeroplane”, “A photo of bicycle”, “A photo of bird”, ... } with each image to do classification.
If the class of the image classified by CLIP is different from its initial class used for generation, we consider this an error sample.
We calculate the error rate on 25K generated data.
As the Tab. 10, we find that we have a 13.79% error rate overall and each class has a different error rate.
This raises a question: How does generation error affect I MAGINARY N ET and can a lower error rate bring better performance?
To answer this question, we randomly sample 5K correct samples and re-train detectors in ISOD.
From the Tab. 11, we surprisedly find the performance of the model gains significantly and obtain the performance of the model trained by 20K un-filtered data.
We explained this as the error data will affect training and the model can learn better detector after filtering generation errors.
[Table]
A. D ISCUSSION OF P REFIX
Prefix is the initial input, which can provide important cues and the language model will extend the prefix to a longer sentence.
Prefix is used in many zero-shot language generation or vision-andlanguage works (Radford et al., 2021; Wang et al., 2021; Cho et al., 2021).
The prefix will provide important cues for language models.
Different prefixes will change the results of the language model and the results of the text-to-image synthesis model then as well.
It will also affect the final performance of detection.
Many works (Radford et al., 2021; Ma et al., 2021; Xu et al., 2022) prove that some simple prefixes can help the multimodal model calculate vision-and-language information better.
Following these previous works, we consider four representative prefixes as follow: (1) “A photo of a { class } ”, (2) “A photograph of a { class } ”, (3) “An image of a { class } ”, and (4) “A { class } ”.
We conduct the comparison on prefixes in Tab. 12.
Each prefix will generate 5K samples for training.
For the Prefix (1) and (2), the language model can generate the sentence that describes the content of images based on the prefix.
Meanwhile, the word in these prefixes, “photo” or “photograph”, forces the text-to-image synthesis model to generate real-style images like photos.
This is also the way for us to control the domain gap between the generated images and real images in the dataset.
Prefix (3) obtains a much worse result.
We can observe that although the language model still can generate the sentence describing objects well, the text-to-image synthesis model generates many images of cartoon or art style.
This leads to a domain gap that makes the model get worse results.
Some classes that have a significant gap in real and art style, such as cat, drop by a huge margin.
Prefix (4) is an even worse prefix than (3).
Most sentences generated by the language model are not describing the object, e.g., “A person should seek professional assistance to obtain an accommodation.”.
We did not train our model based on the prefix (4) because it is hard for the text-to-image synthesis model to generate an image based on such a sentence without any object describing.
[Table]
A. GAN- LIKE L OOP
We setup a GAN like loop by using the detector to find out the samples generated which have the highest loss in a group of samples.
We train I MAGINARY N ET on 10K data first to initialize it and then continue to optimize it with the loop mentioned above on 5K extra samples.
For each loop step, the generator will generate 5 samples, and the detector will choose one sample with the highest loss.
To make a fair comparison
[Table]
we use an I MAGINARY N ET trained with 15K data as the baseline.
From the Tab.
[Table]
we can observe a significant gain by taking into account this GAN like loop.
We think the reason for the improvement is similar to the GAN-based network: the goal of the detector is to minimize the loss of the samples
[Table]
but the goal of the generator is to generate the sample that makes the detector has the highest loss.
These adversarial goals make the network more effective.
Note that, it is hard to optimize the parameter of the text-to-image synthesis model because directly adding the adversarial loss to generative model will make it unstable.
This can be explained by the training task of the text-to-image synthesis model is complex and it is hard to optimize them without their original training data.
Fortunately, because the generator can generate samples infinitely, we can use the detector to select the highest loss sample in a group.
This makes the GAN like loop possible.
A. F AILURE C ASE
From Fig.
8, we can observe that WSOD can better detect objects that are usually accompanying with each other, e.g., rider and horse.
This is because we cannot know all the categories appearing in the generation image precisely.
For example, we cannot know whether the generated images contain only a motorbike or a human riding a motorbike.
If two objects are not accompanying usually, this may not influence the final results because of plenty of training samples.
However, for objects accompanying usually, it is easy for the model to consider them as a whole object.
This may motivate us to explore how to control the accompanying in the generation or better learn accompanying objects.
A modified cross-attentive module for focusing on the key object described in the text may help the model to decrease the possibilities of generating accompanying objects.
[Table]
A. E XPAINATION OF O UTPERFORMING B ASELINE IN ISOD
We think that CLIP cannot use the information from the overlapping proposals because its pretraining is based on image-text matching.
But the detection head of I MAGINARY N ET is designed for the training detection task.
We conduct the extra comparison by fine-tuning the CLIP model on the same data we used for I MAGINARY N ET .
As the Tab. 14, I MAGINARY N ET still outperforms CLIP even after fine-tuning CLIP on our generated data.
But CLIP is still a strong baseline for this task because it uses a large-scale pre-training and can be used to make a comparison in ISOD, the strict setting that models cannot access any real images and annotations.