Original: Bridging the Gap between Object and Image-level Representations for Open-Vocabulary Detection
Revised: Bridging the Gap between Object and Image-level Representations for Open-Vocabulary Detection

Original: Abstract
Revised: Abstract

Original: Existing open-vocabulary object detectors typically enlarge their vocabulary sizesby leveraging different forms of weak supervision.
Revised: Existing open-vocabulary object detectors typically enlarge their vocabulary sizes by leveraging different forms of weak supervision.

Original: This helps generalize tonovel objects at inference.
Revised: This helps generalize to novel objects at inference.

Original: Two popular forms of weak-supervision used in openvocabulary detection (OVD) include pretrained CLIP model and image-level supervision.
Revised: Two popular forms of weak-supervision used in openvocabulary detection (OVD) include pretrained CLIP model and image-level supervision.

Original: We note that both these modes of supervision are not optimally aligned forthe detection task: CLIP is trained with image-text pairs and lacks precise localization of objects while the image-level supervision has been used with heuristics thatdo not accurately specify local object regions.
Revised: We note that both these modes of supervision are not optimally aligned for the detection task: CLIP is trained with image-text pairs and lacks precise localization of objects while the image-level supervision has been used with heuristics that do not accurately specify local object regions.

Original: In this work, we propose to addressthis problem by performing object-centric alignment of the language embeddingsfrom the CLIP model.
Revised: In this work, we propose to address this problem by performing object-centric alignment of the language embeddings from the CLIP model.

Original: Furthermore, we visually ground the objects with onlyimage-level supervision using a pseudo-labeling process that provides high-qualityobject proposals and helps expand the vocabulary during training.
Revised: Furthermore, we visually ground the objects with only imagelevel supervision using a pseudo-labeling process that provides high-quality object proposals and helps expand the vocabulary during training.

Original: We establisha bridge between the above two object-alignment strategies via a novel weighttransfer function that aggregates their complimentary strengths.
Revised: We establish a bridge between the above two object-alignment strategies via a novel weight transfer function that aggregates their complimentary strengths.

Original: In essence, theproposed model seeks to minimize the gap between object and image-centric representations in the OVD setting.
Revised: In essence, the proposed model seeks to minimize the gap between object and image-centric representations in the OVD setting.

Original: On the COCO benchmark, our proposed approachachieves 40.3 AP 50 on novel classes, an absolute 11.9 gain over the previous bestperformance.
Revised: On the COCO benchmark, our proposed approach achieves 36.6 APon novel classes, an absolute 8.2 gain over the previous best performance.

Original: For LVIS, we surpass the state-of-the-art ViLD model by 5.0 mask AP for rare categories and 3.4 overall.
Revised: For LVIS, we surpass the state-of-the-art ViLD model by 5.0 mask AP for rare categories and 3.4 overall.

Original: Our codes will be publicly released.
Revised: Code: https://github.com/hanoonaR/object-centric-ovd .

Original:  Introduction
Revised:  Introduction

Original: Open-vocabulary detection (OVD) aims to generalize beyond the limited number of base classeslabeled during the training phase.
Revised: Open-vocabulary detection (OVD) aims to generalize beyond the limited number of base classes labeled during the training phase.

Original: The goal is to detect novel classes defined by an unbounded(open) vocabulary at inference.
Revised: The goal is to detect novel classes defined by an unbounded (open) vocabulary at inference.

Original: Owing to the challenging nature of the OVD task, different forms ofweak-supervision for novel categories are typically used, e.g. , extra image-caption pairs to enlargethe vocabulary [1], image-level labels on classification datasets [2] and pretrained open-vocabularyclassification models like CLIP [3].
Revised: Owing to the challenging nature of the OVD task, different forms of weak-supervision for novel categories are typically used, e.g. , extra image-caption pairs to enlarge the vocabulary [1], image-level labels on classification datasets [2] and pretrained open-vocabulary classification models like CLIP [3].

Original: The use of weak-supervision to enlarge the vocabulary is intuitiveas the cost of annotating large-category detection datasets is monumental while the image-text/labelpairs are readily available via large classification datasets [4] or internet sources [3, 5].
Revised: The use of weak-supervision to enlarge the vocabulary is intuitive as the cost of annotating large-category detection datasets is monumental while the image-text/label pairs are readily available via large classification datasets [4] or internet sources [3, 5].

Original: One of the major challenges with enlarging vocabulary via image-level supervision (ILS) or pretrainedmodels learned using ILS is the inherent mis-match between region and image-level cues.
Revised: One of the major challenges with enlarging vocabulary via image-level supervision (ILS) or pretrained models learned using ILS is the inherent mis-match between region and image-level cues.

Original: For instance,pretrained CLIP embeddings used in the existing OVD models [6, 2] do not perform well in locatingobject regions [7] since the CLIP model is trained with full scale images.
Revised: For instance, pretrained CLIP embeddings used in the existing OVD models [6, 2] do not perform well in locating object regions [7] since the CLIP model is trained with full scale images.

Original: Similarly, weak supervisionon images using caption descriptions or image-level labels does not convey the precise object-centricinformation.
Revised: Similarly, weak supervision on images using caption descriptions or image-level labels does not convey the precise object-centric information.

Original: For label grounding in images, the recent literature explores expensive pretraining withauxiliary objectives [1] or use heuristics such as, the max-score or max-size boxes [2].
Revised: For label grounding in images, the recent literature explores expensive pretraining with auxiliary objectives [1] or use heuristics such as, the max-score or max-size boxes [2].

Original: In this paper, we set out to bridge the gap between object and image-centric representations within the
Revised: In this paper, we set out to bridge the gap between object and image-centric representations within the

Original: OVD pipeline.
Revised: OVD pipeline.

Original: To this end, we propose to utilize high-quality class-agnostic and class-specific objectproposals via the pretrained multi-modal vision transformer (ViT) [8].
Revised: To this end, we propose to utilize high-quality class-agnostic and class-specific object proposals via the pretrained multi-modal vision transformer (ViT) [8].

Original: The class-agnostic object proposals are then used to distill region-specific information in the CLIP visual embeddings, makingthem suitable for local objects.
Revised: The class-agnostic object proposals are then used to distill region-specific information in the CLIP visual embeddings, making them suitable for local objects.

Original: Furthermore, the class-specific proposal set allows us to visuallyground a larger vocabulary, thereby aiding in generalization to novel categories.
Revised: Furthermore, the class-specific proposal set allows us to visually ground a larger vocabulary, thereby aiding in generalization to novel categories.

Original: Next, the final andimportant question is how to make visual-language (VL) mapping amenable to local object-centricinformation.
Revised: Next, the final and important question is how to make visual-language (VL) mapping amenable to local object-centric information.

Original: For this purpose, we introduce a region-conditioned weight transfer process whichclosely ties together image and region VL mapping.
Revised: For this purpose, we introduce a region-conditioned weight transfer process which closely ties together image and region VL mapping.

Original: In a nut-shell, the proposed approach connectsthe image, region and language representations to generalize better to novel open-vocabulary objects.
Revised: In a nut-shell, the proposed approach connects the image, region and language representations to generalize better to novel open-vocabulary objects.

Original: The major contributions of this work include:
Revised: The major contributions of this work include:

Original: • We propose region-based knowledge distillation to adapt image-centric CLIP embeddings forlocal regions, thereby improving alignment between region and language embeddings.
Revised: • We propose region-based knowledge distillation to adapt image-centric CLIP embeddings for local regions, thereby improving alignment between region and language embeddings.

Original: Weshow that the resulting well-aligned representations aid in improving the overall performanceof our text driven OVD pipeline.
Revised: We show that the resulting well-aligned representations aid in improving the overall performance of our text driven OVD pipeline.

Original: • In order to visually ground weak image labels, our approach performs pseudo-labeling usingthe high-quality object proposals from pretrained multi-modal ViTs.
Revised: • In order to visually ground weak image labels, our approach performs pseudo-labeling using the high-quality object proposals from pretrained multi-modal ViTs.

Original: This helps in enlargingthe class vocabulary and therefore generalizes better to new object classes.•
Revised: This helps in enlarging the class vocabulary and therefore generalizes better to new object classes.

Original:  The above contributions mainly target the visual domain.
Revised: • The above contributions mainly target the visual domain.

Original: In order to preserve the benefits of object-centric alignment in the language domain , we also propose to explicitly condition the (pseudo-labeled) image-level VL mapping on the region-level VL mapping via a novel 
Revised: In order to preserve the benefits of object-centric alignment in the language domain, we also propose to explicitly condition the (pseudo-labeled) image-level VL mapping on the region-level VL mapping via a novel weight transfer function .

Original: weight transfer function .

Original: In this manner, we are the first to simultaneously integrate objectcentric visual and language alignment within a single architecture for OVD.
Revised: In this manner, we are the first to simultaneously integrate objectcentric visual and language alignment within a single architecture for OVD.

Original: • Our extensive experiments demonstrate the improved OVD capability of the proposed approach.
Revised: • Our extensive experiments demonstrate the improved OVD capability of the proposed approach.

Original: On COCO and LVIS benchmarks, our method achieves absolute gains of 11.9 and5.0 AP on novel and rare classes over the current SOTA methods.
Revised: On COCO and LVIS benchmarks, our method achieves absolute gains of 11.9 and 5.0 AP on novel and rare classes over the current SOTA methods.

Original: Further generalizability is demonstrated by our cross-dataset evaluations performed on COCO, OpenImages andObjects365, leading to consistent improvements compared to existing methods.
Revised: Further generalizability is demonstrated by our cross-dataset evaluations performed on COCO, OpenImages and Objects365, leading to consistent improvements compared to existing methods.

Original:  Related Work
Revised:  Related Work

Original: Zero-shot Object Detection (ZSD): This setting involves detecting novel class objects at inference,for which no visual examples are available during training.
Revised: Zero-shot Object Detection (ZSD): This setting involves detecting novel class objects at inference, for which no visual examples are available during training.

Original: Zhu et al.
Revised: Zhu et al.

Original: [9] use semantic informationwith visual features to get proposals for both seen and unseen classes.
Revised: [9] use semantic information with visual features to get proposals for both seen and unseen classes.

Original: Bensal et al.
Revised: Bensal et al.

Original: [10] show thatlearning a good separation between background and foreground is critical in ZSD and propose touse multiple latent classes for modeling background during training.
Revised: [10] show that learning a good separation between background and foreground is critical in ZSD and propose to use multiple latent classes for modeling background during training.

Original: Rahman et al.
Revised: Rahman et al.

Original: [11] propose apolarity loss to solve the ambiguity between background and unseen classes.
Revised: [11] propose a polarity loss to solve the ambiguity between background and unseen classes.

Original: DELO [12] focuses ongenerating good proposals for unseen classes by synthesizing visual features for unseen objects usinga generative model.
Revised: DELO [12] focuses on generating good proposals for unseen classes by synthesizing visual features for unseen objects using a generative model.

Original: Gupta et al.
Revised: Gupta et al.

Original: [13] benefits from the contemporary cues in semantic and visualspace ensuring better class separation for ZSD.
Revised: [13] benefits from the contemporary cues in semantic and visual space ensuring better class separation for ZSD.

Original: Other works use additional learning signals, includingunlabeled images from target domain [14] and raw textual descriptions from the internet [15].
Revised: Other works use additional learning signals, including unlabeled images from target domain [14] and raw textual descriptions from the internet [15].

Original: Although significant progress has been made on this topic [14, 15, 13], the inherent complexity of thetask makes it challenging for the ZSD models to generalize well to unseen object classes.
Revised: Although significant progress has been made on this topic [14, 15, 13], the inherent complexity of the task makes it challenging for the ZSD models to generalize well to unseen object classes.

Original: Weakly-supervised Object Detection (WSOD):
Revised: Weakly-supervised Object Detection (WSOD):

Original: In this setting, only image-level labels are used toapproach object detection [16, 17, 18, 19, 20], or are used alongside the detection dataset to enlargethe detector vocabulary [21, 22, 23].
Revised: In this setting, only image-level labels are used to approach object detection [16, 17, 18, 19, 20], or are used alongside the detection dataset to enlarge the detector vocabulary [21, 22, 23].

Original: Bilen et al.
Revised: Bilen et al.

Original: [24] proposed a weakly-supervised deep detectionnetwork (WSDNN) that uses off-the-shelf region proposals [25, 26] and computes objectness andrecognition scores for each proposal using separate subnetworks.
Revised: [24] proposed a weakly-supervised deep detection network (WSDNN) that uses off-the-shelf region proposals [25, 26] and computes objectness and recognition scores for each proposal using separate subnetworks.

Original: Cap2Det [27] operates in a similarsetting and uses raw text captions to generate pseudo-labels to guide image-level supervision.
Revised: Cap2Det [27] operates in a similar setting and uses raw text captions to generate pseudo-labels to guide image-level supervision.

Original: Li etal.
Revised: Li et al.

Original: [28] uses segmentation-detection collaborative network (SDCN) for accurate detection underweakly-supervised setting using only image labels.
Revised: [28] uses segmentation-detection collaborative network (SDCN) for accurate detection under weakly-supervised setting using only image labels.

Original: PCL [29] proposes to cluster the spatially adjacentproposals and then assign image labels to each cluster.
Revised: PCL [29] proposes to cluster the spatially adjacent proposals and then assign image labels to each cluster.

Original: CASD [30] argues that the detectors trainedonly with image-level labels are prone to detect boxes around salient objects and propose featureattention along with self-distillation to address the issue.
Revised: CASD [30] argues that the detectors trained only with image-level labels are prone to detect boxes around salient objects and propose feature attention along with self-distillation to address the issue.

Original: YOLO

Revised: YOLO

Original: [31] and DLWL [32] augmentsthe detection training by assigning image-level labels to the max-score proposal.
Revised: [31] and DLWL [32] augments the detection training by assigning image-level labels to the max-score proposal.

Original: Detic [2] showsthat using max-size proposal is an optimal choice for assigning image-level labels as it does not relyon the predictions of the network being optimized and provides better signals for the novel classes.
Revised: Detic [2] shows that using max-size proposal is an optimal choice for assigning image-level labels as it does not rely on the predictions of the network being optimized and provides better signals for the novel classes.

Original: We also operate in a similar WSOD setting and use high-quality object proposals from pretrainedmulti-modal ViT [8] to enlarge detector vocabulary and generalize towards novel object categories.ii
Revised: ii

Original: [Figure]
Revised: [Figure]

Original: S p li t
Revised: S p li t

Original: [Figure]
Revised: [Figure]

Original: N o r m
Revised: N o r m

Original: [Figure]
Revised: [Figure]

Original: N o r m
Revised: N o r m

Original: [Figure]
Revised: [Figure]

Revised: We also operate in a similar WSOD setting and use high-quality object proposals from pretrained multi-modal ViT [8] to enlarge detector vocabulary and generalize towards novel object categories.

Original: Open-vocabulary Object Detection (OVD):
Revised: Open-vocabulary Object Detection (OVD):

Original: In OVD, the objective is to detect target class objectsnot present in the training/base class vocabulary.
Revised: In OVD, the objective is to detect target class objects not present in the training/base class vocabulary.

Original: A typical solution of the problem is to replacethe classifier weights with text embeddings of the target vocabulary ( e.g. , GloVe [33], BERT [34],
Revised: A typical solution of the problem is to replace the classifier weights with text embeddings of the target vocabulary ( e.g. , GloVe [33], BERT [34],

Original: CLIP [3]).
Revised: CLIP [3]).

Original: OVR-RCNN [1] uses BERT embeddings as classifier weights and proposes to use openvocabulary captions to learn the vision-to-language mapping.
Revised: OVR-RCNN [1] uses BERT embeddings as classifier weights and proposes to use openvocabulary captions to learn the vision-to-language mapping.

Original: It surpasses the ZSD approaches bya large margin.
Revised: It surpasses the ZSD approaches by a large margin.

Original: ViLD [6] uses pretrained CLIP [3] to distill knowledge into a two-stage objectdetector [35] and replaces the classifier weights with CLIP text embeddings obtained by ensemblingmultiple text prompts ( e.g. , a {category}, a photo of a {category} ).
Revised: ViLD [6] uses pretrained CLIP [3] to distill knowledge into a two-stage object detector [35] and replaces the classifier weights with CLIP text embeddings obtained by ensembling multiple text prompts ( e.g. , a {category}, a photo of a {category} ).

Original: Gao et al.
Revised: Gao et al.

Original: [36] generatepseudo bounding-box labels using pretrained VL models for training open-vocabulary detector.
Revised: [36] generate pseudo bounding-box labels using pretrained VL models for training open-vocabulary detector.

Original: Allthese methods use carefully designed manual prompts for generating text embeddings.
Revised: All these methods use carefully designed manual prompts for generating text embeddings.

Original: DetPro [37]and PromptDet [38] replace these manual prompts with learnable tokens and achieve competitiveresults on novel/rare categories.
Revised: DetPro [37] and PromptDet [38] replace these manual prompts with learnable tokens and achieve competitive results on novel/rare categories.

Original: However, in our work, we use fixed manual prompts and insteadfocus on improving the object-centric representations for open-vocabulary object detection.
Revised: However, in our work, we use fixed manual prompts and instead focus on improving the object-centric representations for open-vocabulary object detection.

Original:  Object-centric Open-Vocabulary Detection
Revised:  Object-centric Open-Vocabulary Detection

Original: Here, we first present a brief overview of the proposed open-vocabulary detection (OVD) framework.
Revised: Here, we first present a brief overview of the proposed open-vocabulary detection (OVD) framework.

Original: As discussed earlier, existing OVD methods use different forms of weak supervision that employimage-centric representations, making them less suited for the end detection task.
Revised: As discussed earlier, existing OVD methods use different forms of weak supervision that employ image-centric representations, making them less suited for the end detection task.

Original: Our proposedmethod aims to bridge the gap between image and object-centric visual-language (VL) representations.
Revised: Our proposed method aims to bridge the gap between image and object-centric visual-language (VL) representations.

Original: We summarize the architectural overview of our method in Fig.
Revised: We summarize the architectural overview of our method in Fig.

Original: The proposed design has three mainelements.
Revised: The proposed design has three main elements.

Original: Our region-based knowledge distillation (refer Sec. 3.2) adapts image-centric languagerepresentations to be object-centric.
Revised: Our region-based knowledge distillation (refer Sec. 3.2) adapts image-centric language representations to be object-centric.

Original: A VL mapping learns to align the local region representations ofthe detector to the language representations by distilling the detector’s region representations withregion representations from a VL model (CLIP).
Revised: A VL mapping learns to align the local region representations of the detector to the language representations by distilling the detector’s region representations with region representations from a VL model (CLIP).

Original: Given weak image-level supervision, we usepseudo-labeling from pretrained multi-modal ViTs (refer Sec. 3.3) to improve generalization of thedetector to novel classes.
Revised: Given weak image-level supervision, we use pseudo-labeling from pretrained multi-modal ViTs (refer Sec. 3.3) to improve generalization of the detector to novel classes.

Original: For efficient combination of the above two proposed components, wecondition the VL mapping learned during the weak supervision on the VL mapping learned withregion-based distillation via a novel weight transfer function (refer Sec. 3.4).
Revised: For efficient combination of the above two proposed components, we condition the VL mapping learned during the weak supervision on the VL mapping learned with region-based distillation via a novel weight transfer function (refer Sec. 3.4).

Original: Specifically, we followa stage-wise learning strategy to first align the region and language embeddings using RKD, and usethis distilled VL mapping for object-centric visual and language alignment in the subsequent stage. 
Revised: Specifically, we follow a stage-wise learning strategy to first align the region and language embeddings using RKD, and use this distilled VL mapping for object-centric visual and language alignment in the subsequent stage. iii

Original: 3.1 Detection Pipeline: Preliminaries
Revised: 3.1 Detection Pipeline: Preliminaries

Original: In the open-vocabulary detection problem, we have access to an object detection dataset where thetraining set, D det , comprises of samples from the set of base object categories, C B .
Revised: In the open-vocabulary detection problem, we have access to an object detection dataset where the training set, D det , comprises of samples from the set of base object categories, C B .

Original: The images ofiii
Revised: The images of

Original: D det are exhaustively annotated with bounding-box labels and corresponding class labels y r ∈ C B ,for the different objects in the image.
Revised: D det are exhaustively annotated with bounding-box labels and corresponding class labels y r ∈ C B , for the different objects in the image.

Original: Given an image I ∈
Revised: Given an image I ∈

Original: R H
Revised: R H

Original: × W × 3 , we design an open-vocabularyobject detector to solve two subsequent problems: (1) effectively localize all objects in the image, (2)classify the detected region into one of the class label of C test , which is provided by the user at testtime.
Revised: × W × 3 , we design an open-vocabulary object detector to solve two subsequent problems: (1) effectively localize all objects in the image, (2) classify the detected region into one of the class label of C test , which is provided by the user at test time.

Original: The categories during test time also include novel categories C N beyond the closed set of basecategories seen during the training phase, i.e. , C test =
Revised: The categories during test time also include novel categories C N beyond the closed set of base categories seen during the training phase, i.e. , C test =

Original: C B ∪ C N .
Revised: C B ∪ C N .

Original: We convert a generic two-stage object detector [35] to an open-vocabulary detector by replacingthe learnable classifier head with fixed language embeddings, T corresponding to the categorynames of C test , that are obtained using a large-scale pretrained VL model.
Revised: We convert a generic two-stage object detector [35] to an open-vocabulary detector by replacing the learnable classifier head with fixed language embeddings, T corresponding to the category names of C test , that are obtained using a large-scale pretrained VL model.

Original: Following [6], we use thetext embeddings from CLIP text encoder [3] for classification, where only the embeddings of C Bcategories, T C B are used during training.
Revised: Following [6], we use the text embeddings from CLIP text encoder [3] for classification, where only the embeddings of C B categories, T C B are used during training.

Original: Specifically, we generate the text embeddings offline, byprocessing the prompts corresponding to each category with a template of ‘ a photo of {category} ’through the CLIP text encoder.
Revised: Specifically, we generate the text embeddings offline, by processing the prompts corresponding to each category with a template of ‘ a photo of {category} ’ through the CLIP text encoder.

Original: The RoI [35] head computes pooled feature representations ϕ ( r )of the proposals r generated by the region proposal network (RPN).
Revised: The RoI [35] head computes pooled feature representations ϕ ( r ) of the proposals r generated by the region proposal network (RPN).

Original: These feature embeddings areprojected to a common feature space shared by the text embedding T using a linear layer f ( · ) , whichwe represent as region embeddings , R = f
Revised: These feature embeddings are projected to a common feature space shared by the text embedding T using a linear layer f ( · ) , which we represent as region embeddings , R = f

Original: ( ϕ ( r ))
Revised: ( ϕ ( r ))

Original: ∈ R D .
Revised: ∈ R D .

Original: For classification, we compute the cosinesimilarity between the region embeddings and text embeddings to find the matching pairs.
Revised: For classification, we compute the cosine similarity between the region embeddings and text embeddings to find the matching pairs.

Original: Duringtraining, the regions that do not match with any of the ground-truths are assigned to the backgroundcategory represented by a fixed all zero embedding.
Revised: During training, the regions that do not match with any of the ground-truths are assigned to the background category represented by a fixed all zero embedding.

Original: We compute the cosine similarity by comparingeach region to each base class, V = sim ( r, b ) = cos (cid:0) R ( r ) , T b (cid:1) ∀ b ∈ C B .
Revised: We compute the cosine similarity by comparing each region to each base class, V = sim ( r, b ) = cos (cid:0) R ( r ) , T b (cid:1) ∀ b ∈ C B .

Original: The classification loss is asoftmax cross-entropy (CE) where the logits are the cosine similarity scores,
Revised: The classification loss is a softmax cross-entropy (CE) where the logits are the cosine similarity scores,

Original: [Equation]
Revised: [Equation]

Original: where τ is the temperature, N is the total number of proposals per image, and r represents a singleproposal with the ground-truth label y r .
Revised: where τ is the temperature, N is the total number of proposals per image, and r represents a single proposal with the ground-truth label y r .

Original: 3.2 Region-based Knowledge Distillation
Revised: 3.2 Region-based Knowledge Distillation

Original: In the OVD setting, we assume that f ( · ) learns a VL mapping and aligns the output region embeddingsof the detector with the corresponding CLIP text embeddings.
Revised: In the OVD setting, we assume that f ( · ) learns a VL mapping and aligns the output region embeddings of the detector with the corresponding CLIP text embeddings.

Original: However, the performance on novelcategories is not comparable to what CLIP encoded embeddings would provide (refer Appendix Cfor details).
Revised: However, the performance on novel categories is not comparable to what CLIP encoded embeddings would provide (refer Appendix B for details).

Original: We hypothesize that this performance gap is mainly due to two reasons, i) the data thathas been used for training CLIP model consists of scene-centric images, making it less suitable forregion classification, e.g., in our case where object-centric tightly bounded proposals are used, ii) thezero-shot generalization ability of the pair-wise trained CLIP image and text embeddings cannot befully utilized due to the mismatch between regions representations from CLIP image encoder and ourdetector.
Revised: We hypothesize that this performance gap is mainly due to two reasons, i) the data that has been used for training CLIP model consists of scene-centric images, making it less suitable for region classification, e.g., in our case where object-centric tightly bounded proposals are used, ii) the zero-shot generalization ability of the pair-wise trained CLIP image and text embeddings cannot be fully utilized due to the mismatch between regions representations from CLIP image encoder and our detector.

Original: Based on these insights, we propose a region-based knowledge distillation (RKD).
Revised: Based on these insights, we propose a region-based knowledge distillation (RKD) .

Original: The proposed RKD uses distillation in the detection pipeline by distilling region embeddings fromhigh-quality class-agnostic proposals ( ˜ r ) obtained from a pretrained multi-modal ViT (MViT) [8].
Revised: The proposed RKD uses distillation in the detection pipeline by distilling region embeddings from high-quality class-agnostic proposals ( ˜ r ) obtained from a pretrained multi-modal ViT (MViT) [8].

Original: Note that we obtain both class-agnostic (used in RKD) and class-specific (refer Sec. 3.3) objectproposals using this pseudo-labeling process, which we refer to as Q pseudo .
Revised: Note that we obtain both class-agnostic (used in RKD) and class-specific (refer Sec. 3.3) object proposals using this pseudo-labeling process, which we refer to as Q pseudo .

Original: This is possible via usingintuitive text queries to interact with the MViT model that can locate generic objects and provides thecorresponding set of candidate proposals.
Revised: This is possible via using intuitive text queries to interact with the MViT model that can locate generic objects and provides the corresponding set of candidate proposals.

Original: The queries can be generic or targeted, based on the task,e.g. , ‘ all objects ’ to generate class-agnostic proposals, or ‘ every dog ’ for a specific class.
Revised: The queries can be generic or targeted, based on the task, e.g. , ‘ all objects ’ to generate class-agnostic proposals, or ‘ every dog ’ for a specific class.

Original: For RKD, we compute class agnostic proposals on D det using simple text query, ‘ all objects ’ andselect top-K proposals (Fig. 3b).
Revised: For RKD, we compute class agnostic proposals on D det using simple text query, ‘ all objects ’ and select top-K proposals (Fig. 3b).

Original: CLIP embeddings I (˜ r ) are then computed offline using the CLIPimage encoder I ( · ) .
Revised: CLIP embeddings I (˜ r ) are then computed offline using the CLIP image encoder I ( · ) .

Original: With the detector region embeddings and the corresponding CLIP regionrepresentations, we propose to use two types of distillation losses to improve the alignment.(1) Point-wise embedding matching loss: The L 1 loss matches the individual region embeddings˜ R =f
Revised: With the detector region embeddings and the corresponding CLIP region representations, we propose to use two types of distillation losses to improve the alignment. (1) Point-wise embedding matching loss: The L 1 loss matches the individual region embeddings ˜ R =

Original:  ( ϕ (˜ r )) with the CLIP region representations I (˜ r ),
Revised: f ( ϕ (˜ r )) with the CLIP region representations I (˜ r )

Original: [Equation]
Revised: [Equation]

Original:  Using this criteria, our visual encoder, along with the VL projection layer f ( · ) , approximates the
Revised: iv Using this criteria, our visual encoder, along with the VL projection layer f ( · ) , approximates the

Original: CLIP image encoder and consequently aligns our region embeddings with the CLIP text embeddings.iv
Revised: CLIP image encoder and consequently aligns our region embeddings with the CLIP text embeddings.

Original: (2) Inter-embedding relationship matching loss (IRM): It is a knowledge distillation based loss L irmthat instills inter-embedding relationships within our region representations to be consistent to the CLIP region representations [39].
Revised: (2) Inter-embedding relationship matching loss (IRM): It is a knowledge distillation based loss L irm that instills inter-embedding relationships within our region representations to be consistent to the CLIP region representations [39].

Original: Instilling such inter-embedding relations would be beneficial as weknow that the teacher model I ( · ) , and the student model (our detector), are different in nature withrespect to their training methods (Fig. 2).
Revised: Instilling such inter-embedding relations would be beneficial as we know that the teacher model I ( · ) , and the student model (our detector), are different in nature with respect to their training methods (Fig. 2).

Original: The IRM loss is defined on pairwise similarity matrices ofthe two different set of embeddings.
Revised: The IRM loss is defined on pairwise similarity matrices of the two different set of embeddings.

Original: Specifically, with the top-K proposals computed from Q pseudo ,we compose K × K similarity matrices for I (˜ r ) and ˜ R denoted by S I and S R respectively.
Revised: Specifically, with the top-K proposals computed from Q pseudo , we compose K × K similarity matrices for I (˜ r ) and ˜ R denoted by S I and S R respectively.

Original: Notably,these matrices are normalized by L2 norm applied row-wise.
Revised: Notably, these matrices are normalized by L2 norm applied row-wise.

Original: The IRM loss is a Frobenius norm∥ · ∥ F , over the mean element-wise squared difference between S I and S R ,[Equation]
Revised: The IRM loss is a Frobenius norm ∥ · ∥ F , over the mean element-wise squared difference between S I and S R ,

Original: [Equation]
Revised: [Equation]

Original: We weight the L 1 and L irm losses by factors β 1 and β 2 , respectively.
Revised: We weight the L 1 and L irm losses by factors β 1 and β 2 , respectively.

Original: Together with the standard twostage detector losses; RPN loss ( L rpn ), regression loss ( L reg ) and classification loss ( L cls ) [35, 40];the overall training objective with RKD can be expressed as,
Revised: Together with the standard twostage detector losses; RPN loss ( L rpn ), regression loss ( L reg ) and classification loss ( L cls ) [35, 40]; the overall training objective with RKD can be expressed as,

Original: [Equation]
Revised: [Equation]

Original: 3.3 Image-level Supervision with Pseudo Box Labels
Revised: 3.3 Image-level Supervision with Pseudo Box Labels

Original: In the open-vocabulary setting, a fundamental challenge is to generalize the detector to novel classes.
Revised: In the open-vocabulary setting, a fundamental challenge is to generalize the detector to novel classes.

Original: However, due to the daunting task of densely locating all objects in natural scenes, the existingdetection datasets are of relatively smaller magnitude compared to the classification datasets, whichare easier to annotate.
Revised: However, due to the daunting task of densely locating all objects in natural scenes, the existing detection datasets are of relatively smaller magnitude compared to the classification datasets, which are easier to annotate.

Original: To this end, Zhou et al.
Revised: To this end, Zhou et al.

Original: [2] proposed to take advantage of large-scale imageclassification dataset during training to expand the detector’s vocabulary.
Revised: [2] proposed to take advantage of large-scale image classification dataset during training to expand the detector’s vocabulary.

Original: However, an importantquestion is how to effectively associate the region proposals of novel objects with the correspondinglabels.
Revised: However, an important question is how to effectively associate the region proposals of novel objects with the corresponding labels.

Original: We note that the existing approach uses heuristics such as, selecting the whole image as asingle box, or just the maximum sized box from the RPN, which can ignore potential objects (Fig. 3a).
Revised: We note that the existing approach uses heuristics such as, selecting the whole image as a single box, or just the maximum sized box from the RPN, which can ignore potential objects (Fig. 3a).

Original: We propose a weakly-supervised method to generalize the detector to novel categories by usingpseudo-box labels from pretrained MViT [8].
Revised: We propose a weakly-supervised method to generalize the detector to novel categories by using pseudo-box labels from pretrained MViT [8].

Original: We follow [2] to train the detector with a combinationof detection and classification dataset.
Revised: We follow [2] to train the detector with a combination v of detection and classification dataset.

Original: A batch of data is prepared by combining data from thev detection dataset D det that are exhaustively annotated with bounding-box and class labels, with datafrom a classification dataset D cls that only contains image-level labels.
Revised: A batch of data is prepared by combining data from the detection dataset D det that are exhaustively annotated with bounding-box and class labels, with data from a classification dataset D cls that only contains image-level labels.

Original: With Q pseudo , we obtain thepseudo-box labels on this classification dataset, which we use for image-level supervision (ILS).
Revised: With Q pseudo , we obtain the pseudo-box labels on this classification dataset, which we use for image-level supervision (ILS) .

Original: Specifically, consider a sample image I ∈ D cls , which has a total of N ground-truth class labels,we generate object proposals offline with the use of MViT corresponding to these weak labels.
Revised: Specifically, consider a sample image I ∈ D cls , which has a total of N ground-truth class labels, we generate object proposals offline with the use of MViT corresponding to these weak labels.

Original: Specifically, we construct N class-specific text queries { t n } Nn =1 with template ‘ every {category} ’,and obtain K proposals { ˜ r k } Kk =1 and corresponding confidence scores { ˜ s k } Kk =1 for each query,[( ˜ r 1 , ˜ s 1 ) , ( ˜ r 2 , ˜ s 2 ) , · · · ( ˜ r K , ˜ s K )]
Revised: Specifically, we construct N class-specific text queries { t n } Nn =1 with template ‘ every {category} ’, and obtain K proposals { ˜ r k } Kk =1 and corresponding confidence scores { ˜ s k } Kk =1 for each query, [( ˜ r 1 , ˜ s 1 ) , ( ˜ r 2 , ˜ s 2 ) , · · · ( ˜ r K , ˜ s K )]

Original: =
Revised: =

Original: Q pseudo ( I, t n ); I ∈ D cls , n ∈ N.
Revised: Q pseudo ( I, t n ); I ∈ D cls , n ∈ N.

Original: We select the top-1 proposal with the highest confidence score, as the pseudo-box label for a particularcategory.
Revised: We select the top-1 proposal with the highest confidence score, as the pseudo-box label for a particular category.

Original: This gives us N high-quality pseudo-box labels for each image, corresponding to its Nimage-level category labels (Fig. 3a).
Revised: This gives us N high-quality pseudo-box labels for each image, corresponding to its N image-level category labels (Fig. 3a).

Original: We compute the region embeddings ˜ R for proposals ˜ r as,
Revised: We compute the region embeddings ˜ R for proposals ˜ r as,

Original: [Equation]
Revised: [Equation]

Original: argmax k

Original: ( ˜ s k ) .

Original: In the case of D det , the training follows the standard two-stage RCNN training recipe.
Revised: In the case of D det , the training follows the standard two-stage RCNN training recipe.

Original: However, for
Revised: However, for

Original: D cls , only the classification loss is updated.
Revised: D cls , only the classification loss is updated.

Original: We call this pseudo-max score , L pms loss.
Revised: We call this pseudo-max score , L pms loss.

Original: [Equation]
Revised: [Equation]

Original: n

Original: BCE ( V , y ˜ r ) , where V = cos (cid:0) ˜ R n , T (cid:1) .

Original: [Equation]

Original: [Equation]

Original: We weight L pms by a factor α and the overall training objective with our ILS can be expressed as,
Revised: We weight L pms by a factor α and the overall training objective with our ILS can be expressed as,

Original: [Equation]
Revised: [Equation]

Original: 3.4 Weight Transfer Function
Revised: 3.4 Weight Transfer Function

Original: To combine the alignment from region-based distillation (Sec. 3.2) with the benefits from weaksupervision with pseudo-box labels (Sec. 3.3), a naive approach would be to train the detector witha combination of losses: L 1 (1), L irm (2) and L pms (4).
Revised: To combine the alignment from region-based distillation (Sec. 3.2) with the benefits from weak supervision with pseudo-box labels (Sec. 3.3), a naive approach would be to train the detector with a combination of losses: L 1 (1), L irm (2) and L pms (4).

Original: However, we demonstrate that a simplecombination of the two approaches does not lead to complimentary benefits, instead they compete witheach other (Table 2).
Revised: However, we demonstrate that a simple combination of the two approaches does not lead to complimentary benefits, instead they compete with each other (Table 2).

Original: The additional supervision from pseudo-labels improves the generalization of thedetector, while the region-based distillation works towards object-centric alignment in the languagedomain, thereby improving the overall performance of the detector.
Revised: The additional supervision from pseudo-labels improves the generalization of the detector, while the region-based distillation works towards object-centric alignment in the language domain, thereby improving the overall performance of the detector.

Original: Our aim is to incorporate thebenefits from the two approaches and preserve the object-centric alignment in the language domain.
Revised: Our aim is to incorporate the benefits from the two approaches and preserve the object-centric alignment in the language domain.

Original: To this end, we use a weight transfer mechanism [41] from VL projection used in region-baseddistillation to the weak supervision by learning a weight transfer function , W T ( · ) .
Revised: To this end, we use a weight transfer mechanism [41] from VL projection used in region-based distillation to the weak supervision by learning a weight transfer function , W T ( · ) .

Original: In other words, the VL projection function f ( · ) used during the weak image-level supervision is explicitly conditioned onthe mapping function used for alignment in the distillation process.
Revised: In other words, the VL projection function f ( · ) used during the weak image-level supervision is explicitly conditioned on the mapping function used for alignment in the distillation process.

Original: This way, both the transformationsare tied together to reinforce mutual representation capability and avoid any conflict in the learnedfunction mapping.
Revised: This way, both the transformations are tied together to reinforce mutual representation capability and avoid any conflict in the learned function mapping.

Original: Let the weights of the projection layer in RKD  and weak image-level supervisionbe represented as W D and W P respectively.
Revised: Let the weights of the projection layer in RKD vi and weak image-level supervision be represented as W D and W P respectively.

Original: The weight transfer operation is given by,
Revised: The weight transfer operation is given by,

Original: [Equation]
Revised: [Equation]

Original: vi Here, W D is kept frozen and we design W T as a 2-layer MLP, W θ 1 followed by W θ 2 a with
Revised: Here, W D is kept frozen and we design W T as a 2-layer MLP, W θ 1 followed by W θ 2 a with

Original: LeakyReLU ( ρ ) activation with a negative slope of 0.1.
Revised: LeakyReLU ( ρ ) activation with a negative slope of 0.1.

Original: Further, we use a skip connection across W Pby projecting the original representations using a separate 2-layer MLP (Fig. 1).
Revised: Further, we use a skip connection across W P by projecting the original representations using a separate 2-layer MLP (Fig. 1).

Original: The total loss here isa combination of L RKD (Eq. 3) and L ILS (Eq. 5) loss, given by,
Revised: The total loss here is a combination of L RKD (Eq. 3) and L ILS (Eq. 5) loss, given by,

Original: L = L rpn + L reg + L cls + β 1 L 1 + β 2 L irm + α L pms .
Revised: [Equation]

Original:  Experiments 
Revised:  Experiments 4.1 Datasets

Original: 4.1 Datasets

Original: [Table]
Revised: [Table]

Original: We conduct our experiments on COCO[42] and LVIS v1.0 [43] under OVD setting.
Revised: We conduct our experiments on COCO [42] and LVIS v1.0 [43] under OVD setting.

Original: For evaluation, we use the generalized ZSD setting where the classifiercontains both base and novel categories.
Revised: For evaluation, we use the generalized ZSD setting where the classifier contains both base and novel categories.

Original: Table 1 summarizes all the datasets usedin our work.
Revised: Table 1 summarizes all the datasets used in our work.

Original: Following [2, 1], we usea subset of ImageNet-21K having 997overlapping LVIS categories and COCOcaptions dataset for ILS in LVIS and COCO experiments respectively (refer Appendix.
Revised: Following [2, 1], we use a subset of ImageNet-21K havingoverlapping LVIS categories and COCO captions dataset for ILS in LVIS and COCO experiments respectively (refer Appendix.

Original: A for moredetails).
Revised: A for more details).

Original: For the pseudo-labeling process Q pseudo , we use the MViT pretrained on a Large-scale Modulated Detection (LMDet) dataset [8].
Revised: For the pseudo-labeling process Q pseudo , we use the MViT pretrained on a Large-scale Modulated Detection (LMDet) dataset [8].

Original: We ensure that MViT pretraining dataset has no overlapwith any of the evaluation datasets in our work.
Revised: We ensure that MViT pretraining dataset has no overlap with any of the evaluation datasets in our work.

Revised: Additionally, in all our experiments we use a pretrained MViT that we train using the author’s provided code on filtered LMDet ( ‡ LMDet) dataset by entirely restricting any exposure to the novel/rare classes in evaluation.

Original: COCO OVD:
Revised: COCO OVD:

Original: We use COCO-2017 dataset for training and validation.
Revised: We use COCO-2017 dataset for training and validation.

Original: We follow the ZS splitsproposed in [10], in which 48 categories are selected as base and 17 are selected as novel classes.
Revised: We follow the ZS splits proposed in [10], in which 48 categories are selected as base and 17 are selected as novel classes.

Original: LVIS OVD: LVIS contains 1203 categories which are further split into frequent, common and rarecategories.
Revised: LVIS OVD: LVIS contains 1203 categories which are further split into frequent, common and rare categories.

Original: Inline with [6, 2], we combine the frequent and common categories to form base classesand keep all rare classes as novel, resulting in 866 base and 337 rare classes.
Revised: Inline with [6, 2], we combine the frequent and common categories to form base classes and keep all rare classes as novel, resulting in 866 base and 337 rare classes.

Original: Cross-transfer Datasets: To validate the adaptability of our method, we evaluate and compareresults of our LVIS trained model on OpenImages[44] and Objects365 [45] and COCO [42] datasets.
Revised: Cross-transfer Datasets: To validate the adaptability of our method, we evaluate and compare results of our LVIS trained model on OpenImages[44] and Objects365 [45] and COCO [42] datasets.

Original: 4.2 Implementation details
Revised: 4.2 Implementation details

Original: We conduct COCO experiments using Faster R-CNN [35] with ResNet-50 backbone.
Revised: We conduct COCO experiments using Faster R-CNN [35] with ResNet-50 backbone.

Original: We train thesupervised-base model on 48 base classes ( C B ) for 1x schedule ( ∼ 12 COCO epochs) and reportbox AP 50 .
Revised: We train the supervised-base model on 48 base classes ( C B ) for 1x schedule ( ∼ 12 COCO epochs) and report box AP 50 .

Original: For RKD, we finetune this model for another 1x schedule using box-labels from C Band class-agnostic proposals from the pretrained MViT [8].
Revised: For RKD, we finetune this model for another 1x schedule using box-labels from C B and class-agnostic proposals from the pretrained MViT [8].

Original: This model is further finetuned for 1xschedule with ILS and the associated weight transfer function using class labels from COCO captionsand corresponding class-specific proposals from MViT.
Revised: This model is further finetuned for 1x schedule with ILS and the associated weight transfer function using class labels from COCO captions and corresponding class-specific proposals from MViT.

Original: This sums to an overall 3x training schedule.
Revised: This sums to an overall 3x training schedule.

Original: For LVIS experiments, we use Mask R-CNN [40] with federated loss [46] and sigmoid cross-entropy,and report mask AP.
Revised: For LVIS experiments, we use Mask R-CNN [40] with federated loss [46] and sigmoid cross-entropy, and report mask AP.

Original: For RKD and weight transfer, we use the same training schedules as of COCOand report the average over three runs.
Revised: For RKD and weight transfer, we use the same training schedules as of COCO and report the average over three runs.

Original: For comparison with Detic [2], we apply our proposed methodon their strong CenterNetV2 [46] baseline under the same settings.
Revised: For comparison with Detic [2], we apply our proposed method on their strong CenterNetV2 [46] baseline under the same settings.

Original: It uses [-ImangeNet
 K pretrainedbackbone-] with 4x schedule using large scale jittering (LSJ) [47] augmentations.
Revised: It uses [-ImangeNet
 K pretrainedbackbone-]ImangeNet21K pretrained backbone with 4x schedule using large scale jittering (LSJ) [47] augmentations.

Original: All of our modelsare trained using 8 A100 GPUs with an approximate training time of 9 and 6 hours for 1x schedule of
Revised: All of our models are trained using 8 A100 GPUs with an approximate training time of 9 and 6 hours for 1x schedule of

Original: COCO and LVIS respectively.
Revised: COCO and LVIS respectively.

Original: In our experiments, we use SGD optimizer with a weight decay of 1 e − 4 and a momentum of 0.9.
Revised: In our experiments, we use SGD optimizer with a weight decay of 1 e − 4 and a momentum of 0.9.

Original: Wetrain for 1x schedule with batch size of 16 and initial learning rate of 0.02 which drops by a factorof 10 at the 8 th and 11 th epoch.
Revised: We train for 1x schedule with batch size of 16 and initial learning rate of 0.02 which drops by a factor of 10 at the 8 th and 11 th epoch.

Original: We set temperature τ to 50.
Revised: We set temperature τ to 50.

Original: Our longer schedules experimentsuse 100-1280 LSJ [47].
Revised: Our longer schedules experiments use 100-1280 LSJ [47].

Original: We use α of 0.1 to weight L pms .
Revised: We use α of 0.1 to weight L pms .

Original: For computing CLIP embeddings weuse the CLIP model ViT-B/32 [3], with input size of 224 × 224.
Revised: For computing CLIP embeddings we vii use the CLIP model ViT-B/32 [3], with input size of 224 × 224.

Original: We use the query ‘ a photo of a{category} ’ for to compute the text embeddings for the classifier.
Revised: We use the query ‘ a photo of a {category} ’ for to compute the text embeddings for the classifier.

Original: For distillation, we use top 5proposals from the pretrained MViT [8] evaluated with generic query, ‘ all objects ’, generatingclass-agnostic proposals.
Revised: For distillation, we use topproposals from the pretrained MViT [8] evaluated with generic query, ‘ all objects ’, generating class-agnostic proposals.

Original: We refer to Appendix E for additional details of the approach we useto generate class-agnostic and class-specific proposals from MViT.
Revised: We refer to Appendix D for additional details of the approach we use to generate class-agnostic and class-specific proposals from MViT.

Original: In COCO experiments, we setweights β 1 and β 2 to 0.15.
Revised: In COCO experiments, we set weights β 1 and β 2 to 0.15.

Original: In LVIS, we set β 1 to 0.15 and β 2 to 0.25.
Revised: In LVIS, we set β 1 to 0.15 and β 2 to 0.25.

Original: We choose these values using arandomized hyperparameter-search on the corresponding held-out datasets.
Revised: We choose these values using a randomized hyperparameter-search on the corresponding held-out datasets.

Original: The 2-layer MLP in ourvii weight transfer function has a hidden dim of 512, and a hidden dim of 1024 is used in the MLP skipconnection across W P in Fig.
Revised: The 2-layer MLP in our weight transfer function has a hidden dim of 512, and a hidden dim of 1024 is used in the MLP skip connection across W P in Fig.

Original: 1 (refer Appendix D for more details).
Revised: 1 (refer Appendix C for more details).

Original: 4.3 Our Approach: Main results
Revised: 4.3 Our Approach: Main results

Original: Table 2 shows the contribution of individual components in our proposed approach.
Revised: Table 2 shows the contribution of individual components in our proposed approach.

Original: Building on top ofthe supervised-base model, RKD shows an absolute gain of 19.9 and 1.2 AP for COCO novel and baseclasses respectively, indicating the adaptability of image-centric CLIP embeddings for local regions.
Revised: Building on top of the supervised-base model, our region-based knowledge distillation (RKD) shows an absolute gain of 19.5 and 1.5 AP for COCO novel and base classes respectively, indicating the adaptability of image-centric CLIP embeddings for local regions.

Original: With ILS, novel class AP improves by 32.5, demonstrating generalization to novel classes and thusenlarging the detector’s vocabulary.
Revised: With pseudo-box labelled weak image-level supervision (PIS), novel class AP improves by 28.7, demonstrating generalization to novel classes and thus enlarging the detector’s vocabulary.

Original: Naively combining the two approaches shows improvement, buthowever struggles to maintain the gains from the individual components.
Revised: Naively combining the two approaches shows improvement, but however struggles to maintain the gains from the individual components.

Original: In contrast, our weighttransfer method suitably combines the complimentary benefits of both components (Fig. 2), achieving40.3 AP on novel classes while maintaining performance on base classes. 
Revised: In contrast, our weight transfer method suitably combines the complimentary benefits of both components (Fig. 2), achieving 36.6 AP on novel classes while maintaining performance on base classes. [Table] 1: Supervised (Base) [Table]

Revised: 2: Base + Region based ditillation (RKD)

Original: [Table]
Revised: [Table]

Revised: 3: Base + ILS with pseudo-box (PIS)

Revised: [Table]

Revised: [Table]

Original: Open-vocabulary Detection - COCO:
Revised: Open-vocabulary Detection - COCO:

Original: We compare our OVD results with previously establishedmethods in Table 3.
Revised: We compare our OVD results with previously established methods in Table 3.

Original: OVR-CNN learns a vision-to-language mapping with expensive pretraining.
Revised: OVR-CNN learns a vision-to-language mapping with expensive pretraining.

Original: Detic uses ILS to improve detection on novel classes.
Revised: Detic uses ILS to improve detection on novel classes.

Original: We use a novel weight transfer function toperform object-centric VL alignment and achieve 54.1 AP on the base classes, surpassing OVR-CNNand Detic by 8.1 AP and 0.3 AP respectively.
Revised: We use a novel weight transfer function to perform object-centric VL alignment and achieve 54.0 AP on the base classes, surpassing OVR-CNN and Detic by 8.0 AP and 0.2 AP respectively.

Original: On novel classes our method achieves 40.3 AP, thehighest novel AP achieved over all methods.
Revised: On novel classes our method achieves 36.6 AP, the highest novel AP achieved over all methods.

Original: In comparison with ViLD, which trains for 8x schedule( ∼ 96 epochs), our method with same schedule provides 56.7 base AP, lagging by 2.8. On novel
Revised: In comparison with ViLD [Table] which trains for 8x schedule ( ∼ 96 epochs), our method with same schedule provides 56.6 base AP, lagging by 2.

Original: [Table]
Revised: [Table]

Original: The results quoted from and our methods are trained for longer 8x schedule (
Revised: [Table] [Table] [Table] [Table]

Original: ‡We train detic for another 1x for a fair comparison with our method.
Revised: [Table] [Table] [Table] [Table]

Original: For ViLD, we use their unified model that trains ViLD-text and ViLD-Image together.
Revised: [Table] [Table] [Table] [Table]

Original: For Detic, we report their best model.
Revised: [Table] viii

Original: 294 classes, we achieve 40.5 AP surpassing ViLD by a gain of 12.9.
Revised: On novel classes, we achieve 36.9 AP surpassing ViLD by a gain of 9.3.

Original: In contrast to ViLD design, ourweight transfer function allows both RKD and ILS to provide complimentary gains without anycompetition among the two methods [6].
Revised: In contrast to ViLD design, our weight transfer function allows both RKD and ILS to provide complimentary gains without any competition among the two methods [6].

Original: Open-vocabulary Detection - LVIS:
Revised: Open-vocabulary Detection - LVIS:

Original: Table 4 (left) compares our results with ViLD [6] on LVISbenchmark.
Revised: Table 4 (left) compares our results with ViLD [6] on LVIS benchmark.

Original: With 3x training schedule ( ∼ 36 epochs) we perform reasonably well compared to ViLD32x schedule ( ∼ 384 epochs), already surpassing the rare AP by 1.1 while having slightly lowerperformance on frequent class.
Revised: With 3x training schedule ( ∼ 36 epochs) we perform reasonably well compared to ViLD 32x schedule ( ∼ 384 epochs), already surpassing the rare AP by 1.1 while having slightly lower performance on frequent class.

Original: Extending our model to 8x schedule fills the gap, surpassing ViLD by0.8 in frequent and 5.0 AP in rare classes respectively.
Revised: Extending our model to 8x schedule fills the gap, surpassing ViLD by 0.8 in frequent and 5.0 AP in rare classes respectively.

Original: In Table 4 (right), we compare our method with
Revised: In Table 4 (right), we compare our method with

Original: Detic using their strong LVIS baseline that uses CenterNetV2 network.
Revised: Detic using their strong LVIS baseline that uses CenterNetV2 network.

Original: Following similar settings,we finetune their box-supervised model using our weight transfer method and show improvements.viii
Revised: Following similar settings, we finetune their box-supervised model using our weight transfer method and show improvements. [Table]

Original: [Table]
Revised: [Table]

Revised: Strict Open-vocabulary Setting: Inspired from

Revised: Detic [2], we define our work under the weaklysupervised open-vocabulary setting as it uses imagelevel labels for expanding the detector’s vocabulary.

Revised: However in this setting, the complete target vocabulary set is not known, i.e. , in LVIS, we use ILS from

Revised: ImageNet-21K for some base and novel categories.

Revised: To evaluate our model in a strict open-vocabulary setting, we modify our ILS by considering a very large vocabulary.

Revised: Specifically, we expand the vocabulary to five times its size, by using ILS from randomly sampled 5K categories from ImageNet-21k, in addition to the LVIS base classes.

Revised: Table 5 compares our strict OVD setting results with ViLD [6].

Revised: Different from the above two settings, we also train our LVIS model under stricter OVD conditions in a non weakly supervised setting by only using LVIS base categories for ILS.

Revised: We achieve an overall 21.71 AP which is similar to the model trained using ILS from 997 categories (22.75 AP).

Original: [Table]
Revised: [Table]

Original: Cross-dataset evaluation performance: We provide cross-dataset evaluation of our model in Table 5and compare with prior OVD works.
Revised: Cross-dataset evaluation performance: We provide cross-dataset evaluation of our model in Tableand compare with prior OVD works.

Original: ViLD and Detic-base[2] are box-supervised baseline models for ViLD and Detic respectively.
Revised: ViLD-text[6] and Detic-base[2] are box-supervised baseline models for ViLD and Detic respectively.

Original: Our methodbuilds on top of Detic-base and shows favourableresults when directly transferred to cross-datasetswithout any dataset-specific finetuning.
Revised: Our method builds on top of Detic-base and shows favourable results when directly transferred to cross-datasets without any dataset-specific finetuning.

Original: We useour method trained on LVIS and report AP 50 on
Revised: We use our method trained on LVIS and report AP 50 on

Original: COCO [42], OpenImages [44] and Objects365 [45].
Revised: COCO [42], OpenImages [44] and Objects365 [45].

Original: We refer to Appendix B for qualitative results.

Original: 4.4 Analysis of RKD and ILS
Revised: 4.4 Analysis of RKD and ILS

Original: Effect of Region-based Knowledge Distillation (RKD):[Table] We ablate the effect of L 1 (Eq. 1) and L irm(Eq. 2) RKD approach on COCO (Table 6).
Revised: Effect of Region-based Knowledge Distillation (RKD): We ablate the effect of L 1 (Eq. 1) and L irm (Eq. 2) RKD approach on COCO (Table 7).

Original: Theresults show the importance of both loss functions, where using L 1 loss over base model withtop-5 proposals from MViT [8] improves thebase and novel class by 1.9 and 15.0 AP (row-1vs 3).
Revised: The results show the importance of both loss functions, where using L 1 loss over base model with top-5 proposals from MViT [8] improves the base and novel class by 1.9 and 15.0 AP (row-1 vs 3).

Original: Using L irm in row-4 further improves theoverall and novel class AP.
Revised: Using L irm in row-4 further improves the overall and novel class AP.

Original: To show the importance of using quality proposals in RKD, we compare the model trained with L 1 loss using [-top-
-] RPN vs MViT proposals (row-2 vs 3).
Revised: To show the importance of using quality proposals in RKD, we compare the model trained with L 1 loss using [-top-
-]top-5 RPN vs MViT proposals (row-2 vs 3).

Original: All the models in rows 2-4 are finetuned on the base model.
Revised: All the models in rows 2-4 are finetuned on the base model.

Original: Effect of Weak Image-level Supervision (ILS):
Revised: Effect of Weak Image-level Supervision (ILS):

Original: [Table]

Original: We compare different choices of ILS in Table 7.
Revised: We compare different choices of ILS in Table 8.

Original: Our L pms loss (Eq. 4) is compared with previously adopted ILS approaches [31, 32, 2] (rows2-3).
Revised: Our L pms loss (Eq. 4) is compared with previously adopted ILS approaches [31, 32, 2] (rows 2-3).

Original: In row-4, we generate class-agnostic object proposals using ‘ all objects ’ text querywith multi-modal ViTs (MViTs) [8] and selectmax-size proposal for ILS.
Revised: In row-4, we generate class-agnostic object proposals using ‘ all objects ’ text query with multi-modal ViTs (MViTs) [8] and select max-size proposal for ILS.

Original: In row-5, our proposed ILS approach uses  target specific ‘ every{category} ’ text query with MViT and selectstop-1 proposal for each ILS category.
Revised: In row-5, our proposed ILS approach uses ix [Table] [Table] target specific ‘ every {category} ’ text query with MViT and selects top-1 proposal for each ILS category.

Original: Our method (row-5) shows better performance compared toother alternatives.
Revised: Our method (row-5) shows better performance compared to other alternatives.

Original: Additionally, we present all ablations on LVIS dataset in Appendix D.
Revised: Additionally, we present all ablations on LVIS dataset in Appendix C.

Revised:  Qualitative Results

Original:  Conclusion
Revised:  Conclusion

Original: This paper develops a novel framework to leverage the representation and generalization capability ofpre-trained multi-modal models towards improved open-vocabulary detection (OVD).
Revised: This paper develops a novel framework to leverage the representation and generalization capability of pre-trained multi-modal models towards improved open-vocabulary detection (OVD).

Original: Specifically,we note that the existing OVD methods use weak supervision modes that are more image-centric,rather than object-centric for the end detection task.
Revised: Specifically, we note that the existing OVD methods use weak supervision modes that are more image-centric, rather than object-centric for the end detection task.

Original: We proposed a novel knowledge distillationapproach together with object-level pseudo-labeling to promote region-wise alignment between visualand language representations.
Revised: We proposed a novel knowledge distillation approach together with object-level pseudo-labeling to promote region-wise alignment between visual and language representations.

Original: Our weight transfer module provide an integration mechanism tocombine the benefits of knowledge distillation and object-level pseudo-labeling.
Revised: Our weight transfer module provide an integration mechanism to combine the benefits of knowledge distillation and object-level pseudo-labeling.

Original: We demonstrateencouraging results on a four popular OVD benchmarks, demonstrating sound generalization ability.ix
Revised: We demonstrate encouraging results on a four popular OVD benchmarks, demonstrating sound generalization ability.

Revised: Acknowledgements: We acknowledge the compute support provided at the Swedish National

Revised: Infrastructure for Computing (SNIC), partially funded by the Swedish Research Council through grant agreement 2018-05973.

Revised: x

Original: Checklist
Revised: Checklist

Original: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Original: [Yes] The abstract and introduction clearly reflects the main contributions and scope of the paper.
Revised: [Yes] The abstract and introduction clearly reflects the main contributions and scope of the paper.

Original: (b) Did you describe the limitations of your work?
Revised: (b) Did you describe the limitations of your work?

Original: [Yes] We have discussed the limitations of our work.
Revised: [Yes] We have discussed the limitations of our work.

Original: Please refer to Appendix F. (c) Did you discuss any potential negative societal impacts of your work?
Revised: Please refer to Appendix E. (c) Did you discuss any potential negative societal impacts of your work?

Original: [Yes] Please refer to Appendix G. (d) Have you read the ethics review guidelines and ensured that your paper conformsto them?
Revised: [Yes] Please refer to Appendix F. (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes] We have read the ethics review guidelines and discussed the ethical implications of our work in Appendix H.
Revised: [Yes] We have read the ethics review guidelines and discussed the ethical implications of our work in Appendix G. 2.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A]
Revised: [N/A]

Original: Our findings and propositions are mainly based on experiments and empirical results.
Revised: Our findings and propositions are mainly based on experiments and empirical results.

Original: However, we have added relevant mathematical information in our theoretical formulations.
Revised: However, we have added relevant mathematical information in our theoretical formulations.

Original: (b) Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [N/A] 
Revised: [N/A] 3.

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)?
Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [Yes] We provide thecode along with the instructions to reproduce our main experiments in the supplemental 
Revised: [Yes] We provide the code along with the instructions to reproduce our main experiments in the supplemental material.

Original: material.
Revised: xiii

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: [Yes] We have provided all the training details including the data splitsand hyperparameter choices in our paper.
Revised: [Yes] We have provided all the training details including the data splits and hyperparameter choices in our paper.

Original: Please refer to sections 4.1 and 4.2.
Revised: Please refer to sections 4.1 and 4.2.

Original: (c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)?
Revised: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [No] Due to the limited availability of compute resources, we have not reported these statistics.
Revised: [No] Due to the limited availability of compute resources, we have not reported these statistics.

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or provided
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: xii
Revised: [Yes] Yes we have provided these details in the main paper. Please refer to the section 4.2. 4.

Original:  (a) If your work uses existing assets, did you cite the creators?
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] We have cited all relevent existing works and assets which are related/used in our work.
Revised: [Yes] We have cited all relevent existing works and assets which are related/used in our work.

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [Yes] We provide license details of the assets used in our work.
Revised: [Yes] We provide license details of the assets used in our work.

Original: Please refer to section I. (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: Please refer to section H. (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [Yes] We provide our code for reproducing main experiments of our work in the supplemental material.
Revised: [Yes] We provide our code for reproducing main experiments of our work in the supplemental material.

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?

Original: [N/A] We use publically available datasets for our experiments.We
Revised: [N/A] We use publically available datasets for our experiments.

Original:  have not explicitly discussed such consent in the main paper, but we have checked and made sure that all used datasets are allowed to be used for research.
Revised: We have not explicitly discussed such consent in the main paper, but we have checked and made sure that all used datasets are allowed to be used for research.

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?

Original: [Yes] We discuss this in the supplemental material.
Revised: [Yes] We discuss this in the supplemental material.

Original: If you used crowdsourcing or conducted research with human subjects...[Table] (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
Revised: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [N/A]

Original: [Table]

Original:  (b) Did you describe any potential participant risks, with links to Institutional Review[Table] Board (IRB) approvals, if applicable?[N/A]
Revised: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Original: [Table] (c) Did you include the estimated hourly wage paid to participants and the total amount[Table] spent on participant compensation?
Revised: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: [N/A]
Revised: [N/A]

Original: xiii
Revised: xiv