GLIPv2: Unifying Localization and VL Understanding
Abstract
We present GLIPv2, a grounded VL understanding model, that serves both localization tasks (e.g., object detection, instance segmentation) and Vision-Language (VL) understanding tasks (e.g., VQA, image captioning).
GLIPv2 elegantly unifies localization pre-training and Vision-Language Pre-training (VLP) with three pre-training tasks: phrase grounding as a VL reformulation of the detection task, region-word contrastive learning as a novel region-word level contrastive learning task, and the masked language modeling.
This unification not only simplifies the previous multi-stage VLP procedure but also achieves mutual benefits between localization and understanding tasks.
Experimental results show that a single
GLIPv2 model (all model weights are shared) achieves near SoTA performance on various localization and understanding tasks.
The model also shows (1) strong zero-shot and few-shot adaption performance on open-vocabulary object detection tasks and (2) superior grounding capability on VL understanding tasks. Code is released at https://github.com/microsoft/GLIP .
Introduction
Recently, a general interest arises in building general-purpose vision systems [24, 28, 66, 47], also called vision foundation models [6, 67], that solve various vision tasks simultaneously, such as image classification [35], object detection [44] and Visual-Language (VL) understanding [3, 11, 32].
Of particular interest, is the unification between localization tasks (e.g., object detection [44] and segmentation [8, 23]) and VL understanding tasks (e.g., VQA [3] and image captioning [11]).
Localization pre-training benefits VL tasks [1, 70], and the “localization->VLP” two-stage pretraining procedure [46, 57, 13, 56, 39, 37, 75, 42, 40] is the common practice in VL community.
A long-standing challenge is the unification of localization and understanding, which aims at mutual benefit between these two kinds of tasks, simplified pre-training procedure, and reduced pre-training cost.
However, these two kinds of tasks appear to be dramatically different: localization tasks are visiononly and require fine-grained output (e.g., bounding boxes or pixel masks), while VL understanding tasks emphasize fusion between two modalities and require high-level semantic outputs (e.g., answers or captions).
[24, 28, 66] have made early attempts at unifying these tasks in a straightforward multi-task manner, where a low-level visual encoder is shared across tasks, and two separate high-level branches are designed for localization and VL understanding, respectively.
The localization tasks are still vision-only and do not benefit from the rich semantics in vision-language data.
As a result, such unified models see the marginal mutual benefit or even performance degradation [28] compared with task-specific models. [Figure]
In this paper, we identify “VL grounding” as a “meta”-capability for localization and understanding capabilities.
VL grounding involves not only understanding an input sentence but also localizing the mentioned entities in the image (see an example in Figure 1).
We build a grounded VL understanding model (GLIPv2) as a unified model for localization and VL understanding tasks.
Localization + VL understanding = grounded VL understanding .
Localization tasks involve both localization and semantic classification, where classification can be cast as a VL understanding problem using the classification-to-matching trick (Section 3.1).
Therefore, we reformulate localization tasks as VL grounding tasks, in which the language input is a synthesized sentence as the concatenation of category names [41].
Localization data are turned into VL grounding data, accordingly.
The massive VL understanding data (image-text pairs) can be easily turned into VL grounding data in a self-training manner [41].
Therefore, GLIPv2 has a unified pre-training process: all task data are turned into grounding data and GLIPv2 is pre-trained to perform grounded VL understanding.
A stronger VL grounding task: inter-image region-word contrastive learning .
GLIP [41] proposes the phrase grounding task as its pre-training task, which we argue is an easy task and does not fully utilize data information.
For example, in the VL grounding task in Figure 1, the phrase grounding task only requires the model to match a given image region to one of the three phrases in the text input, i.e., “green, pink striped, or plain white umbrella?”.
This 1-in-3 choice is very easy, only requires color understanding, but loses lots of information in this grounding data: the umbrellas are not any other colors, like black, yellow, etc; objects in those regions are umbrellas but not any other categories, like car, bike, etc.
From a contrastive learning view, this phrase grounding task only has two negatives.
More negatives can be created from this annotation and thus enable stronger contrastive learning.
In GLIPv2, we introduce the novel inter-image region-word contrastive learning task, which leverages phrases from other sentences in the same batch as potential negatives, as another much stronger VL grounding task.
This new region-word contrastive loss enables GLIPv2 to learn more discriminative region-word features and demonstrates improvements over all downstream tasks.
GLIPv2 achieves mutual benefit between localization and VL understanding .
1) Experimental results (Table 2) show that a single GLIPv2 model (all model weights are shared) achieves near SoTA performance on various localization and understanding tasks.
Thanks to semantic-rich annotations from the image-text data, GLIPv2 shows superior zero-shot and few-shot transfer learning ability to open-world object detection and instance segmentation tasks, evaluated on the LVIS dataset and the "Object Detection in the Wild (ODinW)" benchmark. 3)
GLIPv2 enables language-guided detection and segmentation ability, and achieves new SoTA performance on the Flick30K-entities phrase grounding and PhraseCut referring image segmentation tasks.
4) Inherently a grounding model,
GLIPv2 leads to VL understanding models with strong grounding ability, which are self-explainable and easy to debug.
For example, GLIPv2, when GLIPv2 is finetuned on VQA, it can answer questions while localizing mentioned entities (see Figure 1 and Section 4.4).
Related Work
Localization models.
Traditionally, localization tasks such as object detection and segmentation are single-modality and output bounding boxes or pixel masks [52, 43, 26, 14, 53, 10, 9].
One challenge of these single-modality models lies in generalization to rare and novel concepts: it is hard to collect localization data that cover many rare categories [23].
A long line of research focuses on this generalization problem, under the name of zero-shot [4, 76, 7, 77], weakly-supervised [19, 5, 61], or open-vocabulary [68, 22] localization.
Built upon MDETR [30] and GLIP [41], GLIPv2 converts localization tasks into a grounded vision-language task using the classification-to-matching trick (Section 3).
Thus GLIPv2 can learn from the semantic-rich vision-language data and shows strong performance on open-vocabulary localization tasks.
Vision-language understanding models.
Vision-language (VL) understanding tasks such as VQA [3], image captioning [11], and image-text retrieval [31] involve understanding visual semantics and how they are expressed in natural language.
Many VL models (e.g., BUTD) [2, 70] rely on a pre-trained localization model as their visual encoder; the downside is the pro-longed “localization->VLP” pre-training pipeline [46, 57, 13, 56, 39, 37, 75, 42, 40].
In contrast, GLIPv simplifies the pre-training pipeline and enables grounded VL understanding for better interpretability (Section 4.4).
Unifying localization and understanding.
[24, 28, 66] made pioneering efforts in unifying localization and understanding.
However, localization tasks are still treated as single-modality tasks, while
VL tasks involve two modalities.
The unification is achieved via straightforward multi-tasking: a low-level visual encoder is shared across tasks and two separate branches are designed for localization and VL understanding.
Such unified models do not bring evident mutual benefit and often underperform task-specific models.
In contrast, GLIPv2 identifies grounded VL understanding as a meta-task for localization and understanding.
The task unification brings architecture unification: the unified grounded VL understanding model empowers a localization branch with VL capacity, arriving at a unified branch that excels at both tasks.
GLIPv2 vs GLIP.
1) GLIP shows that grounded pre-training improves localization.
GLIPv2 further shows grounded pre-training improves VL understanding and thus leads to a unified model for localization and VL understanding.
2) GLIPv2 introduces the inter-image region-word contrastive loss, which is another and stronger grounding task than the pre-training task in GLIP.
The proposed loss can be viewed as a region-word level generalization of the prevalent image-level contrastive learning [38, 51, 65]. 3) GLIPv2 outperforms GLIP on all benchmarks with the same pre-training data.
GLIPv2: Unifying Localization and VL Understanding
Based on the reformulation of object detection as a generalized phrase grounding task in GLIP [41], we unify both localization and VL understanding tasks as grounded vision-language tasks.
A grounded vision-language task takes both image and text as inputs, and outputs region-level understanding results (e.g., detection, segmentation) and/or image-level understanding results with associated grounding/localization information (e.g., VQA, image captioning).
We will present the unified grounded VL formulation and architecture in Section 3.1, the pre-training losses in Section 3.2, and transfer to downstream tasks in Section 3.3.
3.1 A Unified VL Formulation and Architecture
At the center of GLIPv2’s unified formulation is the classification-to-matching trick, which reformulates any task-specific fixed-vocab classification problem as an task-agnostic open-vocabulary vision-language matching problem.
The best example is the reformulation of image classification as image-text matching in CLIP [51], which enables the model to learn from raw image-text data directly, and achieves strong zero-shot results on open-vocabulary classification tasks.
In GLIPv2, we replace every semantic classification linear layer in traditional single-modality vision models with a vision-language matching dot-product layer.
As illustrated in Figure 1, GLIPv2’s unified VL architecture is based on the generic architecture we term Architecture Π .
It consists of a dual encoder, denoted as Enc V and Enc L , and a fusion encoder, denoted as Enc V L .
The model takes an image-text pair ( Img , Text ) as input, and extract visual and text features as below:
[Equation]
where (˚ O , ˚ P ) and ( O, P ) denote the image/text features before and after VL fusion, respectively.
Vision-Language understanding tasks.
Arch Π is the most popular model architecture for VL understanding tasks.
Given the cross-modality fused representations O and P , it is straightforward to add lightweight task-specific heads for various VL tasks.
For example, GLIPv2 adds a two-layer
MLP on top of text features P as the masked language modeling (MLM) head, to perform the MLM pre-training.
We provide model details of VQA and image captioning in Section 3.3. (Language-guided) object detection and phrase grounding.
Following GLIP [41], GLIPv2 uses the classification-to-matching trick to unify detection and grounding.
More specifically, for detection, we simply replace the class logits S cls =
OW T , where W is the weight matrix of the box classifier, with a task-agnostic region-word similarity logits S ground =
OP T , where text features P are label embeddings from a task-agnostic language encoder.
As shown in Figure 1, object detection and phrase grounding share the same input/output format and model architecture.
See GLIP [41] for more details.
Their only difference is the input text format: (1) for object detection, the text input is a string of concatenated candidate object labels; (2) for phrase grounding, the text input is a natural language sentence.
We refer to GLIP [41] for more details. (Language-guided) instance segmentation and referring image segmentation.
Given the object detection results, an instance segmentation head is added to classify each pixel within the box into a semantic class.
Again, GLIPv2 uses the classification-to-matching trick to produce a unified instance segmentation head for the standard instance segmentation tasks and the referring image segmentation tasks and leverage both types of data for its pre-training.
This classification-to-matching trick can also apply to many other semantic classification heads in single modality CV models (e.g., semantic segmentation) and thus transfers them to language-guided CV models.
3.2 GLIPv2 Pre-training
The GLIPv2 is pre-trained with three pre-training losses: phrase grounding loss L ground from a vision-language reformulation of the object detection task, region-word contrastive loss L inter from a novel region-word level contrastive learning task, and the standard masked language modeling loss
L mlm proposed in BERT [17].
[Equation]
Similar to losses in detection tasks, the grounding loss L ground has two parts: the localization loss L loc trains localization heads with bounding-box supervision, e.g., RPN loss, box regression loss and/or centerness loss [59]; the intra-image region-word alignment loss L intra is essentially the semantic classification/retrieval loss for each region.
Intra-image region-word alignment loss.
Given one image-text pair ( Img , Text ) , we obtain the image and text features after cross-modality fusion O and P .
The Intra-image region-word alignment loss is computed by
[Equation]
where OP T is the similarity score between image regions and word tokens, and T is the target affinity matrix determined by the ground-truth annotations.
The loss function loss is typically a cross-entropy loss for two-stage detectors [53] and a focal loss [43] for one-stage detectors.
However, as discussed in Section 1, this intra-image region-word contrastive learning is rather weak in the sense of contrastive learning, due to the limited number of phrases that can one caption can contain.
GLIP [41] alleviates this problem by appending a few negative sentences to form a longer text input with more (negative) phrases.
However, constrained by the maximal length of text tokens (256 in GLIP and GLIPv2), only a few negative sentences can be added and the number of negative phrases remains in the order of 10’s.
This small-negative-example problem also exists in detection data [41] when the input text cannot include all class names in a detection dataset, e.g., Objects365.
Inter-image region-word contrastive loss.
In GLIPv2, we propose using phrases from other imagetext pairs in the same batch as negative examples, which effectively increases the number of negative examples to the order of 1000’s, with nearly negligible additional computational cost.
As in (1), given a batch of image-text pairs ( Img i , Text i ) Bi =1 and their ground-truth annotations ( T i )
Bi
=1 , the model produces the image and text features before and after VL fusion, denoted as (˚ O i , ˚ P i )
Bi
=1 and ( O i , P i ) Bi
=1 , respectively.
Then as illustrated in Figure 2 (Left), a batch-wise similarity matrix S batchground and a batch-wise target affinity matrix T batch are constructed by considering all the image regions and text phrases across this batch.
Their ( i, j ) ’
th blocks are obtained as below:
[Equation]
if i
[Equation]
j obtained by label propagation , otherwise.
[Equation]
The inter-image region-word contrastive loss is then defined as the standard bi-directional contrastive loss applied on all image regions and phrases in this batch:
L
[Equation]
cross_entropy_loss ( S batchground , T batch , axis = 0)+ cross_entropy_loss ( S batchground , T batch , axis = 1) . Compared with that in the inter-image contrastive loss (3), the number of negatives is multiplied by batch size B in this inter-image contrastive loss (5).
We elaborate two important details in (4).
GLIPv2 uses the image text features (˚ O i , ˚ P i )
Bi
=1 before VL fusion, not ( O i , P i )
Bi
=1 after VL fusion, to compute the batch-wise similarity matrix in the inter-image contrastive loss (4).
Otherwise, the image and text features after VL fusion would have seen the paired information (1), and thus the model can easily rule out the negatives from misaligned images/texts.
(2) We cannot simply assign all regions and texts from unpaired image-text as negative pairs, as done in the standard contrastive loss in CLIP [51].
Instead, we determine the off-diagonal blocks in the target affinity matrix T batch by label propagation .
For example, as illustrated in Figure 2 (Left), if a region is annotated as “person”, it should be a positive pair with all “person” phrases in detection-type texts.
We do not propagate positives to grounding-type texts (natural sentences) because phrases in sentences carry contexts that are unique to that image-sentence pair.
Pre-training with both detection and paired-image-text data.
GLIPv2 pre-training data is in the image-text-target triplet format ( Img , Text , T ) , where the target affinity matrix T contains the box-label localization annotations.
We also use massive image-text pair data ( Img , Text ) to pre-train
GLIPv2, by generating grounding boxes ˆ T for phrases in the text with the GLIP pre-trained model from [41].
The human-annotated OD/grounding data provides high-fidelity localization supervision, while the massive image-text data greatly improves the concept diversity for GLIPv2.
Second-stage pre
-training of the segmentation head.
GLIPv2 performs a second-stage pre-training of the language-guided segmentation head on both instance segmentation and image referring segmentation data, while fixing all other parts of the model.
3.3 Transfer GLIPv2 to Localization and VL Tasks
We introduce two ways to easily transfer GLIPv2 to various downstream tasks.
In addition, GLIPv can perform conventional VL tasks (e.g., VQA) along with localization, effectively making every task we consider a “grounded VL understanding” task.
One model architecture for all.
GLIPv2 can be transferred to downstream tasks by fine-tuning the model with an (optional) task-specific head.
For detection and segmentation tasks, no task-specific head is needed as the pre-training architecture can inherently perform detection and segmentation.
For VL tasks: for VQA, a classification head is added on top of the hidden representation of the start-of-sequence token; for caption generation, we train with a unidirectional language modeling loss, which maximizes the likelihood of the next word given context.
We use a unidirectional attention mask and prevent the image part from attending to the text in the fusion layers. 0  0: Inter -image Contrastive Loss Hat. Person…Frisbee.
: Intra - image Contrastive Loss
Annotated positives Propagated positives
Pers on Bicyc le Hair #dryer. Person Hat
Fris #bee.
DeepFusionBlock
[Figure]
[Figure]
[Figure]
Person Bicycle Hair #dryer.
Person Hat Fris #bee. One set of weights for all. There is a growing interest in developing models that can be transferred to various tasks while only changing the least amount of parameters to save training time and storage cost [55, 36].
Following GLIP, GLIPv2 can be transferred to localization tasks in a zero-shot or a prompt-tuning setting (Section 4.2).
One single GLIPv2 model can serve various tasks, where each task only keeps few or no parameters.
Of particular interest is the prompt tuning setting.
For a certain localization task, the text prompt is the same for all input images; thus, we could directly tune ˚ P , a small prompt embedding matrix, to adapt GLIPv2 to new tasks.
Prompt tuning in a deep-fused model such as GLIPv2 is different from the conventional linear probing/prompt tuning setting [62, 51, 73] in shallow-interacting vision models such as CLIP.
The latter can also be viewed as only tuning a small prompt/softmax embedding P ; however, tuning P only affects the very last layer of the model while the visual representation is still frozen.
In contrast, GLIP/GLIPv2’s visual representation is conditioned on the prompt embedding ˚ P ; tuning ˚ P changes the text, visual, as well as fused embeddings.
As a result, prompt tuning in GLIPv2 is highly effective, often matching the performance of fine-tuning (see Table 2).
This is in contrast to the common observation in CV that linear probing lags behind fine-tuning by a large gap [25].
Grounded VL understanding.
GLIPv2 also enables grounded VL understanding, where we retain the ability to perform grounding when fine-tuning the model to a downstream VL task.
This increases the interpretability of the model.
Specifically, we first turn the VL data of the downstream task into grounded VL data using a pre-trained GLIP model.
Then we train the model with both the downstream task head and grounding head.
For VQA, the model is trained to predict the answer and ground entities in the question as well as the implied entity in the answer; for captioning, the model is trained to predict the next word given the context and ground the current decoded word.
By tuning localization tasks into a grounded VL task and augmenting VL tasks with grounding ability, we effectively turn every task into a grounded VL understanding task (see examples in Figure 1).
Experiments
In this section, we show that GLIPv2 serves as a performant and easy-to-deploy general-purpose vision system.
1) One Model Architecture for All (Section 4.1).
GLIPv2 can be directly fine-tuned to both localization and VL understanding tasks with minimal architecture change.
It achieves performance on par with SOTA models with specialized architectures.
2) One Model Weight for
All (Section 4.2).
GLIPv2 can be transferred to localization tasks in a zero-shot manner with zero [Table] For LVIS, we report the and on minival to avoid we parameter update; with prompt tuning, a single GLIPv2 model can achieve comparable performance with fully fine-tuned settings on both localization and understanding tasks.
Following GLIP [41], we adopt Swin Transformer [45] as the image encoder Enc V , text transformers [60, 51] as the text encoder Enc L , Dynamic Head [15] with language-aware deep fusion [41] as the fusion encoder Enc V L , and Hourglass network [49] as instance segmentation head feature extractor.
We train GLIPv2 at three scales: GLIPv2-T, GLIPv2-B, and GLIPv2-H.
GLIPv2-T has the same model config and initialization as GLIP-T: Swin-Tiny and BERT-Base as the dual encoder.
The model is pre-trained on the following data: 1) O365, 2)
GoldG as in
GLIP-T (C), and 3) Cap4M, 4M image-text pairs collected from the web with boxes generated by
GLIP-T [41].
GLIPv2-B/GLIPv2-H are based on Swin-Base/Swin-Huge and the pre-layernorm text transformer [18] as dual encoder, and are initialized from the UniCL [65] checkpoints.
We observe much stabler training with GPT-type pre-layernorm transformer [18] than BERT-type post-layernorm transformer.
The training data contain: 1) FiveODs (2.78M data) 1 ; 2) GoldG as in MDETR [30]; and 3) CC15M+SBU, 16M public image-text data with generated boxes by GLIP-L [41].
Segmentation heads of GLIPv2 models are pre-trained on COCO, LVIS [23] and PhraseCut [63], with all other model parameters are frozen.
Note All datasets above were collected by the creators (cited) and consent for any personally identifiable information (PII) was ascertained by the authors where necessary.
Due to limited space, we refer to supplementary for details of training recipes and hyper-parameters.
4.1 One Model Architecture for All
We compare GLIPv2 to existing object detection and vision-language pre-training methods on a wide range of tasks.
We fine-tune the model on 8 different downstream tasks and report the performance in
Table 1.
We make the following observations.
GLIPv2 v.s. specialized Localization methods.
GLIPv2 outperforms previous localization models on generalization to both common and rare classes and domains with a single model architecture and pre-training stage .
1) OD on common categories (COCO-Det) , GLIPv2-T achieves 5.8 improvement compared to the standard DyHead-T trained on O365 (55.5 v.s. 49.7).
GLIPv2-H reaches 62.4 AP on test-dev, and surpass the performance of the previous SoTA model GLIP-L. 2) OD on rare / unseen categories (LVIS) , GLIPv2-T outperforms a supervised MDETR on the bbox by a great margin (59. v.s. 24.2).
3) Generalization to diverse real-word tasks (ODinw) , GLIPv2-T (55.5) performs better than original GLIP-T (64.9) on the average of 13 public datasets; GLIPv2-B outperforms GLIP-L by 0.5 AP.
4) Instance segmentation (COCO-Mask & PhraseCut) , for traditional instance segmentation [Table] [Figure] [Figure]
[Table]
[Figure] [Table] [Figure] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table] exhibit superior data efficiency. (i.e., COCO-Mask), GLIPv2-H outperforms the well-known Mask R-CNN by a great margin on segm . For language-guided segmentation (i.e., PhraseCut), compared to MDETR, GLIPv2-T achieves an improvement of 5.7 mask AP.
GLIPv2 v.s. specialized VL Understanding methods.
GLIPv2 rivals with SoTA specialized models for VL tasks.
1) For VQA , GLIPv2 outperforms VisualBERT and UNITER and approaches the previous SoTA model VinVL. 2)
For Captioning , the best GLIPv2 even surpasses VinVL (VinVL and GLIPv2 are not trained with CIDEr optimization).
GLIPv2 v.s. localization and VL models.
Prior works such GPV, UniT and Unicorn have also explored unifying localization and VL models (see a discussion in Section 2).
GLIPv2 outperforms all previous systems on both localization and VL tasks.
For the best GLIPv2-H, it outperforms the
UniT by a great margin (18.3 AP) on COCO object detection tasks.
Meanwhile, it also surpasses
UniT’s performance on VQA by 6.9 points and GPV’s peformance on Image Captioning as well.
Takeaway.
Most notably, GLIPv2 outperforms previous “unified” models (GPV, UniT, MDETR,
Unicorn) by a large margin.
This is the first time that a single model architecture could achieve near SoTA performance on both localization and understanding.
In contrast, in prior work, there exists certain trade-off between localization and understanding: models that aim to achieve high understanding performance tend to have lower localization performance (e.g., UNiT’s detection performance is limited to the DETR [9] architecture), as it is not trivial to merge a SoTA localization branch and a SoTA VL branch into a single model.
4.2 One Set of Model Parameters for All
GLIPv2 is pre-trained to perform grounding; thus it can be transferred to various localization tasks with changing zero or few parameters.
We evaluate GLIPv2 under two such settings: 1) direct evaluation, where we transfer the model “as is” without any parameter change, and 2) prompt tuning, where only the prompt embedding is tuned for specific tasks (Section 3.3).
Direct evaluation.
The pre-trained GLIPv2 can be directly evaluated on any object detection task (by concatenating the object categories into a text prompt) and visual grounding task without any further tuning.
We evaluate the models on four localization tasks: COCO, ODinW, LVIS, and Flickr30, and their results are presented in Table 2.
Note that for GLIPv2-B and GLIPv2-H, the training sets of Flick30K and LVIS are present in the pre-training data.
Thus, reported numbers on these metrics are not zero-shot evaluation (we have marked them gray).
For all other evaluation results, the models are evaluated in zero-shot settings without any further tuning.
GLIPv2 can be effortlessly transferred to different localization tasks without further tuning.
For
COCO , GLIPv2-T achieves a zero-shot performance of 47.3 without seeing any COCO training images.
This surpasses well-established supervised systems (e.g., Mask R-CNN) and also outperforms
GLIP-T by 0.7 AP.
For ODinW , GLIPv2 also shows strong zero-shot performance.
GLIPv2-T (48.5) surpasses the GLIP-T (46.5).
Meanwhile, the zero-shot performance of GLIPv2-B and GLIPv2H even surpasses the 10-shot tuning performance of DyHead-T (to be introduced in Figure 3).
For LVIS , GLIPv2-T achieves a 3 AP improvement performance compared to the GLIP-T. 4) For
Flickr30K , GLIPv2-B achieves even higher number (87.2) compared to original GLIP-L (87.1).
Prompt Tuning.
Following GLIP, GLIPv2 supports efficient prompt tuning: the visual representation is heavily conditioned on the text representation due to the deep fusion block (Section 3.3); thus we could fine-tune only the prompt embedding for each task but still maintain high performance.
Prompt tuning GLIPv2 achieves similar performance as full fine-tuning.
When comparing the performance of each task in Table 1 and 2 at the same time, for GLIPv2, prompt tuning performance almost matches the one model architecture results on localization tasks, without changing any of the grounding model parameters.
4.3 GLIPv2 as a Strong Few-Shot Learner
We demonstrate GLIPv2’s performance on ODinW datasets with respect to different amounts of training data in Figure 3.
The performance improvement between GLIPv2-T and GLIP-T exhibits more superior data efficiency for prompt tuning.
We compare with the SoTA detector DyHead-T, pre-trained on Objects365 in Table 3.
It can be seen that a zero-shot GLIPv2-T (48.5) outperforms a outperforms 5-shot DyHead-T (46.4) while the performance of one-shot GLIPv2-H (61.3) surpasses a all-shot fully supervised DyHead-T (60.8).
4.4 Analysis
Pre-training losses Table 4 shows the performance of the downstream tasks with different variants of our method. Compared to the GLIP pre-training tasks with only intra-image region-word contrastive loss (Row 3), adding inter-image word-region loss (Row 5) substantially improves the pre-trained model performance across all the object detection tasks (COCO, ODinW, and LVIS) on both zero-shot and fine-tuned manner. Consistent with common observations from most VL understanding methods, adding MLM loss (Row4) benefits for learning the representation for understanding tasks (Flick30k, VQA, and Captioning).
Furthermore, using all three losses together at the 1st stage pre-training and doing the 2nd stage pre-training without MLM on OD and GoldG data, GLIPv2 (Row6) can perform well on both the localization and VL understanding tasks. An additional stage of pre-training is applied for small models (GLIPv2-T and GLIPv2-B) due to limited model capacity. In order to achieve higher performance on both localization and understanding tasks, we find that including all data (even with some noise) and MLM loss in the first stage of pre-training will benefit the model for learning a better representation of both localization and understanding capability.
Since the OD tasks require the model with more accurate localization ability, in our 2nd stage of pre-training, we decide to eliminate the MLM loss. The large model (GLIPv2-H) does not need this additional stage because it has enough capacity to learn both wordregion alignment and MLM together in a single stage. Pre-training data Table 5 reports the last checkpoint results on GLIPv2 when we do the scaling up of pre-training data. As more weak image-text pair data (Cap) is involved in our training, it benefits both standard/in-domain (i.e., COCO, Flickr30K) and large-domain gap (i.e., ODinW, LVIS) tasks.
We also show that by adding the inter-image region-word contrastive helps when we are fixing the data at the same scale.
For large-domain gap tasks, adding the inter-image region-word contrastive
[Table]
[Table] [Table] loss will further boost the model to learn better representation. For more detailed scaling-up effects on various tasks under all the checkpoints for GLIP and GLIPv2, refer to Appendix.
Note that the ( Img , Text , T ) data used in GLIPv2 pre-training can be just human-annotated data (Row1&2 in Table 5), with which GLIPv2 pre-training does not involve any pseudo data from a pre-trained grounding/localization model.
In order to achieve the best performance, GLIPv2 uses image-text pair data with pseudo boxes (Cap) from a pre-trained GLIP model (Row3-6 in Table 4), which is trained with the same "grounded VL understanding" task but just with smaller data.
Grounded Vision-Language Understanding GLIPv2 can be trained to perform a VL task and grounding at the same time (Section 3.3).
We denote such an ability as grounded VL understanding.
In Figure 1, we showcase grounded predictions of GLIPv2 on VQA and COCO captions.
We also conduct quantitative evaluations (Table 6).
The model achieves strong performance for both VL understanding (on COCO Caption) and localization (on Flickr30K Grounding).
Such an ability to produce high-level semantic outputs (i.e., answers and captions) and supporting localization results is another appealing trait of GLIPv2, as potential users can have a better understanding of the model behaviour.
See more detailed analysis and qualitative examples in the Appendix.
Conclusion and Social Impacts
This paper proposes GLIPv2, a unified framework for VL representation learning that serves both localization tasks and VL understanding tasks.
We experimentally verify the effectiveness of the unified model and the novel region-word contrastive learning. Compared to existing methods,
GLIPv2 achieves competitive near SoTA performance on various localization and understanding tasks. However, additional analysis of the data and the model is necessary before deploying it in practice since large-scale web data may contain unintended private information, unsuitable images/text, or some bias leakage. Further investigation may be needed for web data due to the above issues.  Acknowledgement
We thank anonymous reviewers for their comments and suggestions. Additional thanks go to the Microsoft Research Horizontal AI Team and Microsoft Alexander Multi-modal Team for providing computer resources for large-scale training. The baseline models used in our experiments are based on the open-source code released in the GitHub repository; we acknowledge all the authors who made their code public, which tremendously accelerates our project progress.
The checklist follows the references.
Please read the checklist guidelines carefully for information on how to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or [N/A] .
You are strongly encouraged to include a justification to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] • Did you include the license to the code and datasets?
[No] The code and the data are proprietary.
• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] (c) Did you discuss any potential negative societal impacts of your work?
[Yes] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] (b) Did you mention the license of the assets?
[Yes] (c) Did you include any new assets either in the supplemental material or as a URL? [No] (d) Did you discuss whether
and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
Appendix
The appendix is organized as follows: • In Section A, we provide more visualizations of our model’s predictions on various localization and VL understanding tasks. • In Section B, we describe all our evaluated tasks and their dataset in detail.
• In Section C, we discuss the difference between our additional inter-image region-word contrastive loss and some other well-known losses that were also applied over a full batch in multiple works.
• In Section D, we introduce the training details and hyperparameters used in Section 4 in the main paper. • Section E, we analyze the effect of using different language encoder and their pre-trained weights in our models. • In Section F, we provide more results for all the checkpoints of adding pre-training data (refer to Section 4 in the main paper). • In Section G, we provide a detailed analysis of the experiments of grounded captioning (mentioned in Section 4 in the main paper).
• In Section H, we give out a comparison for the model’s inference speed.
• In Section I, we clearly provide the original sources of the images that are used in our paper.
• In Section J, we present per-dataset results for all experiments in ODinW.
[Figure]
A Visualization
We provide a clearer illustration of GLIPv2 in Figure 4, which elegantly unifies various localization (object detection, instance segmentation) and VL understanding (phrase grounding, VQA and captioning) tasks.
More visualizations of the predictions under various tasks from GLIPv2 are also provided to indicate the model’s strength and capability.
Please refer to Figure 5 for OD / Grounding, Figure 6 for Instance / Referring Image Segmentation, and Figure 7 for Grounded VL Understanding.
[Figure]
B Tasks and dataset descriptions B.1 (Language-guided) object detection and phrase grounding
COCO.
[8] The Microsoft Common Objects in Context dataset is a medium-scale object detection dataset.
It has about 900k bounding box annotations for 80 object categories, with about 7.
annotations per image.
It is one of the most used object detection datasets, and its images are often used within other datasets (including VG and LVIS).
Flickr30k-entities.
[50] Given one or more phrases, which may be interrelated, the phrase grounding task is to provide a set of bounding boxes for each given phrase.
We use the Flickr30k-entities dataset for this task, with the train/val/test splits as provided by [41] and evaluate our performance in terms of Recall.
Flickr30K is included in the gold grounding data so we directly evaluate the models after pre-training as in MDETR [30].
We predict use any-box protocol specified in MDETR.
ODinW. We use 13 datasets from Roboflow 2 .
Roboflow hosts over 30 datasets, and we exclude datasets that are too challenging (e.g., detecting different kinds of chess pieces) or impossible to solve without specific domain knowledge (e.g., understanding sign language).
We provide the details of the 13 datasets we use in Table 7.
We include the PASCAL VOC 2012 dataset as a reference dataset, as public baselines have been established on this dataset.
For PascalVOC, we follow the convention
[Figure]
[Figure]
[Table]
and report on the validation set.
For Pistols, there are no official validation or test sets so we split the dataset ourselves.
B.2 (Language-guided) instance segmentation and referring image segmentation
LVIS.
The Large Vocabulary Instance Segmentation dataset has over a thousand object categories, following a long-tail distribution with some categories having only a few examples. Similar to VG, LVIS uses the same images as in COCO, re-annotated with more object categories. In contrast to
COCO, LVIS is a federated dataset, which means that only a subset of categories is annotated in each image.
Annotations, therefore, include positive and negative object labels for objects that are present and categories that are not present, respectively.
In addition, LVIS categories are not pairwise disjoint, such that the same object can belong to several categories.
PhraseCut.
[63] Besides object detection, we show that our GLIPv2 can be extended to perform segmentation by evaluating the referring expression segmentation task of the recent PhraseCut[63] which consists of images from VG, annotated with segmentation masks for each referring expression.
These expressions comprise a wide vocabulary of objects, attributes and relations, making it a challenging benchmark.
Contrary to other referring expression segmentation datasets, in PhraseCut the expression may refer to several objects and the model is expected to find all the corresponding instances.
B.3 VQA and image captioning
VQA.
[20] requires the model to predict an answer given an image and a question.
We conduct experiments on the VQA2.0 dataset, which is constructed using images from COCO.
It contains 83k images for training, 41k for validation, and 81k for testing.
We treat VQA as a classification problem with an answer set of 3,129 candidates following the common practice of this task.
For our best models, we report test-dev and test-std scores by submitting to the official evaluation server.
COCO image captioning.
[11] The goal of image captioning is to generate a natural language description given an input image.
We evaluate GLIPv2 on COCO Captioning dataset and report
BLEU-4, CIDEr, and SPICE scores on the Karparthy test split.
C Difference between inter-image region-word contrastive loss with other "region-word" losses.
As far as we know, up to the deadline (05/19/2022) for NeurIPS submission, there are only three published papers (VILD [21], RegionCLIP [72], and X-VLM [69]) that have the flavor of "region-
[Table]
word" loss applied over full batch.
We discuss the difference between our work and the three aforementioned works in the following:
All these three works use “region-sentence" loss, i
e., the similarity between a region feature and the [CLS] token of a sentence, instead of true "region-word" loss used in GLIPv2.
As a result, none of these three works made use of the phrase grounding data, which may contain multiple entities in one sentence during their training.
It is the most important point in GLIPv2 to use phrase grounding data and pseudo grounding data to train a unified grounded VL understanding model.
GLIPv2 has carefully designed the positive label propagation in our inter-image region-word contrastive loss to mitigate the wrong assumption that "every unpaired region-word pair is negative".
As far as we know, no previous work has mentioned this mechanism of positive label propagation before.
There are some other differences.
For example, in VILD, its “region-sentence loss" is actually not a contrastive loss over full-batch but a classification loss over a fixed vocabulary per sample (see the definition of L V iLD − text ) .
Upon all three points above, we believe that our inter-image region-word contrastive loss is novel and has a significant difference from previous works.
D Training details and hyperparamters D.1 Pre-training
Pre-training data.
There are three different types of data in pre-training 1) detection data 2) grounding data 3) caption data, as shown in Table 8.
The detection data includes Object365 [54],
COCO [8], OpenImages [33], Visual Genome [34], and ImageNetBoxes [16].
The grounding data includes GoldG, 0.8M human-annotated gold grounding data curated by MDETR [30] combining
Flick30K, VG Caption, and GQA [29].
The Cap4M is a 4M image-text pairs collected from the web with boxes generated by GLIP-T(C) in [41], and CC (Conceptual Captions)
+ SBU (with 1M data).
Implementation details.
In Section 4 in the main paper, we introduced GLIPv2-T, GLIPv2-B,
GLIPv2-H, and we introduce the implementation details in the following.
We pre-train GLIPv2-T based on Swin-Tiny models with 32 GPUs and a batch size of 64.
We use a base learning rate of 1 × 10 − 5 for the language backbone (BERT-Base) and 1 × 10 − 4 for all other parameters.
The learning rate is stepped down by a factor of 0.1 at the 67% and 89% of the total 330,000 training steps.
We decay the learning rate when the zero-shot performance on COCO saturates.
The max input length is 256 tokens for all models.
To optimize the results for object detection, we continue pre-training without the MLM loss for another 300,000 steps.
We pre-train GLIPv2-B based on Swin-Base models with 64 GPUs and a batch size of 64.
We use a base learning rate of 1 × 10 − 4 for all parameters, including the language backbone (CLIP-type pre-layernorm transformer).
The learning rate is stepped down by a factor of 0.1 at the 67% and 89% of the total 1 million training steps.
We decay the learning rate when the zero-shot performance on COCO saturates.
The max input length is 256 tokens for all models.
To optimize the results for object detection, we continue pre-training without the MLM loss for another 500,000 steps.
We pre-train GLIPv2-H based on the CoSwin-Huge model from Florence [67] with 64 GPUs and a batch size of 64.
We use a base learning rate of 1 × 10 − 4 for all parameters, including the language backbone (CLIP-type pre-layernorm transformer).
The learning rate is stepped down by a factor of
[Figure]
0.1 at the 67% and 89% of the total 1 million training steps.
We decay the learning rate when the zero-shot performance on COCO saturates.
The max input length is 256 tokens for all models.
We found that there is no need to continue pre-training without MLM loss for the huge model.
Mask heads of GLIPv2-T, GLIPv2-B and GLIPv2-H are pre-trained COCO, LVIS and PhraseCut, while freezing all the other model parameters.
This mask head pre-training uses batch size 64, and goes through COCO for 24 epochs, LVIS for 24 epochs, and PhraseCut for 8 epochs, respectively.
GLIPv2 uses Hourglass network [49] as instance segmentation head feature extractor, and utilizes the "classification-to-matching" trick to change the instance segmentation head linear prediction layer (outputs K -dimensional logits on each pixel) to a dot product layer between pixel visual features and the word features after VL fusion.
GLIPv2-T and GLIPv2-B use a very basic Hourglass network for segmentation head feature extractor: only 1 scale and 1 layer, with hidden dimension 256.
GLIPv2-H uses a larger Hourglass network for segmentation head feature extractor: 2 scales and 4 layers, with hidden dimension 384.
D.2 Downstream tasks
OD / Grounding.
When fine-tuning on COCO, we use a base learning rate of 1 × 10 − 5 andtraining epochs for the pre-trained GLIPv2-T model, and a base learning rate of 5 × 10 − 6 andtraining epochs for the pre-trained GLIPv2-B and GLIPv2-H models.
For direct evaluation on LVIS, since LVIS has over 1,200 categories and they cannot be fit into one text prompt, so we segment them into multiple chunks, fitting 40 categories into one prompt and query the model multiple times with the different prompts.
We find that models tend to overfit on
LVIS during the course of pre-training so we closely monitor the performance on minival for all models and report the results with the best checkpoints in Table 2 in the main paper.
For direct evaluation on Flickr30K, models may also overfit during the course of pre-training so we monitor the performance on the validation set for all models and report the results with the best checkpoints in Table 2 in the main paper.
Instance segmentation /
Referring Image Segmentation.
Given the pre-trained model with pretrained mask head, we simply fine-tune the entire network to get the task-specific fine-tuned models.
For fine-tuning on COCO instance segmentation, we use a base learning rate of 1 × 10 − 5 andtraining epochs for the pre-trained GLIPv2-T model, and a base learning rate of 5 × 10 − 6 andtraining epochs for the pre-trained GLIPv2-B and GLIPv2-H models.
For fine-tuning on LVIS instance segmentation, we use a base learning rate of 1 × 10 − 5 andtraining epochs for the pre-trained GLIPv2-T model, and a base learning rate of 5 × 10 − 6 andtraining epochs for the pre-trained GLIPv2-B and GLIPv2-H models.
For fine-tuning on PhraseCut Referring Image segmentation, we use a base learning rate of 1 × 10 −and 12 training epochs for the pre-trained GLIPv2-T model, and a base learning rate of 5 × 10 − 6 and 3 training epochs for the pre-trained GLIPv2-B and GLIPv2-H models.
(Grounded) VQA.
To fine-tune GLIPv2 for VQA, we feed the image and question into the model and then take the output feature sequence P from the language side (after the VL fusion) and apply a ‘attention pooling’ layer to obtain a feature vector P vqa .
More specifically, the attention pooling layer applies a linear layer followed by softmax to obtain normalized scaler weights, and then these weights are used to compute a weighted sum to produce the feature vector p vqa .
This feature vector is then fed to a 2-layer MLP with GeLU activation [27] and a final linear layer to obtain the logits for the 3129-way classification.
4 Following standard practice [58], we use binary cross entropy loss to take account of different answers from multiple human annotators.
Following VinVL [71], we train on the combination of train2014 + val2014 splits of the VQAv2 dataset, except for the reserved 2k dev split.
For the ablation studies we report the accuracy on this 2k dev split.
Other than the conventional VQA setting, we also experimented a new ‘grounded VQA’ setup, which the model is required to not only predict the answer, but also ground the objects (predict bounding boxes in the image) mentioned in the question and answer text, see Figure 8(iii).
Note that the language input is the question appended by a [MASK] token, and this [MASK] token should ground to the object if the answer is indeed an object in the image.
The total training loss is summing the grounding loss (intra-image region-word contrastive loss) and the VQA loss described previously.
(Grounded) Image Captioning.
We fine-tune the pre-trained model on COCO Caption “Karpathy” training split.
The training objective is uni-directional Language Modeling (LM), which maximizes the likelihood of the next word at each position given the image and the text sequence before it.
To enable autoregressive generation, we use uni-directional attention mask for the text part, and prevent the image part from attending to the text part in the fusion layers.
Although the training objective (LM) is different from that in pre-training (i.e., bi-directional MLM), we directly fine-tune the model for image captioning to evaluate its capability of generalizing to VL generation tasks.
Our model is trained with cross entropy loss only, without using CIDEr optimization.
For grounded image captioning (Figure 8), we add the grounding loss (intra-image region-word contrastive loss) in training, which is calculated in the same way as in pre-training.
We use Flickr30K training split for this task.
During inference, for each predicted text token, we get its dot product logits with all the region representations and choose the maximum as the associated bounding box.
E Analysis on the effect of different language encoders and pre-trained weights
For GLIPv2-T, we use the ImageNet pre-trained Swin-Transformer to initialize the image encoder and
BERT-base-uncased to initialize the language encoder.
For GLIPv2-B, we use the pre-trained paired image-language encoder from UniCL (CLIP-like pre-training, https://github.com/microsoft/
UniCL ) for initialization.
We did an ablation study on the different language encoders (UniCL vs.
BERT)
and found that their results are nearly the same, as shown in Figure 9.
Therefore, UniCL initialization does not skew the good localization performance.
The main reason for us to keep the UniCL(CLIP-like) language encoder is due to its Pre-LayerNorm [64] operation.
We find the
UniCL(CLIP-like) language encoder with Pre-LayerNorm is more stable during the training compared with BERT, which uses Post-LayerNorm.
F More analysis on pre-training data
Table 5 in the main paper reports the last checkpoint results on GLIPv2 when we do the scaling up of pre-training data.
As more weak image-text pair data (Cap) is involved in our training, it benefits both standard/in-domain (i.e., COCO, Flickr30K) and large-domain gap (i.e., ODinW, LVIS) tasks.
Further adding the inter-image region-word contrastive helps when we are fixing the data at the same scale.
For large-domain gap tasks, adding the inter-image region-word contrastive loss will further boost the model to learn better representation. To learn more scaling-up effects on various tasks under all the checkpoints for GLIP and GLIPv2, see Figure 10. Given the considerable amount of improvement of GLIPv2 when the number of caption data increases from 0M to 12M, we hypothesize that it has potential to further grow by training on even larger-scale web image-text pairs. G Experiments on grounded image captioning
The grounded captioning task requires the model to generate an image caption and also ground predicted phrases to object regions.
The final predictions consist of (1) the text captions (2) predicted object regions, and (3) the grounding correspondence between the phrases and regions.
Following the established benchmarks [48, 74], we evaluate the caption metrics on COCO Captions and report the grounding metrics on Flick30K, as shown in Table 9.
[Table]
H Inference speed
We test the inference speed for GLIPv2 on V100 with batch size 1 and show its comparison to
MDETR, as shown in Table 10.
[Figure]
I Figure Reference
We provided the original sources of the images that are used in our paper in the following.
All datasets above were collected by the creators (cited) and consent for any personally identifiable information (PII) was ascertained by the authors where necessary.
Figure 1 in the main paper - The top left and the bottom middle figures are the 281759.jpg in COCO val set; The left right images are (from top to down: (1) 2588.jpg in ODinW Aquarium test set.
(2) 13923.jpg in LVIS val set.
(3) 132690.jpg in VQA2.0 val set (question id is 132690002).
(4) 462565.jpg in COCO Caption val set.
[Table]
Figure 2 in the main paper - The top left figure is the 209297.jpg in COCO train set; The bottom left is the
Figure 4 in the Appendix - Same as Figure 1.
The top left and the bottom middle figures are the 281759.jpg in COCO val set
Figure 5 in the Appendix - Row 1 (from left to right): (1) jpg val set val
Figure 6 in the Appendix - Row 1 (from left to right): (1) jpg val set
J All results for ODinW
We report the per-dataset performance under 0,1,3,5,10-shot and full data as well as prompt tuning, and full-model tuning in Table 11 and Table 12 (on the next page)
[Table] [Table]