Coarse-to-Fine Vision-Language Pre-training with Fusion in the Backbone
Abstract
Vision-language (VL) pre-training has recently received considerable attention.
However, most existing end-to-end pre-training approaches either only aim to tackle VL tasks such as image-text retrieval, visual question answering (VQA) and image captioning that test high-level understanding of images, or only target region-level understanding for tasks such as phrase grounding and object detection.
We present F IBER ( F usion- I n-the- B ackbone-based transform ER ), a new VL model architecture that can seamlessly handle both these types of tasks.
Instead of having dedicated transformer layers for fusion after the uni-modal backbones, F IBER pushes multimodal fusion deep into the model by inserting cross-attention into the image and text backbones, bringing gains in terms of memory and performance.
In addition, unlike previous work that is either only pre-trained on image-text data or on fine-grained data with box-level annotations, we present a two-stage pretraining strategy that uses both these kinds of data efficiently: ( i ) coarse -grained pre-training based on image-text data; followed by ( ii ) fine -grained pre-training based on image-text-box data.
We conduct comprehensive experiments on a wide range of VL tasks, ranging from VQA, image captioning, and retrieval, to phrase grounding, referring expression comprehension, and object
detection.
Using deep multimodal fusion coupled with the two-stage pre-training, F IBER provides consistent performance improvements over strong baselines across all tasks, often outperforming methods using magnitudes more data.
Code is available at https://github.com/microsoft/FIBER .
Introduction
Inspired by the success of language model pre-training [14, 56, 46], coupled with the unification of architectures used in the NLP and computer vision communities [15, 6], vision-language pre-training (VLP) [68, 49, 37, 8] has been receiving an increasing amount of attention.
It has been proven that
VLP can establish state-of-the-art performance on visual question answering [4], visual reasoning [66], image captioning, and image-text retrieval [45].
The pre-training objectives commonly used for these tasks, such as image-text matching, image conditioned masked language modeling and image-text constrastive learning, require multimodal understanding at the image level.
Typically, this means the pre-training is done using images at lower resolution ( e.g. , 384 × 384), making it possible to scale up training by using large batch sizes.
Recently, it has also been shown that tasks such as image classification and object detection (OD), which have been traditionally viewed as vision-only tasks, can benefit from being cast as VL tasks [55, 28, 38, 29].
Inspired by MDETR [29], GLIP [38] reformulates standard classificationbased OD as phrase grounding.
This opens up the possibility to leverage VLP for OD, and vice versa, and this unification has led to impressive performance on several established OD as well as phrase grounding benchmarks [54].
Since these tasks involve fine-grained image understanding between regions in the image and phrases in the text, and also require prediction of precise bounding boxes at the output, the pre-training typically involves using high resolution input images ( e.g. , 800 × 1,333).
Existing multimodal architectures typically do not support both kinds of tasks.
Specifically, the fully end-to-end VLP models such as ALBEF [36], METER [16], and SimVLM [75] can achieve the state of the art (SoTA) on image-level understanding tasks, but it is non-trivial to extend them for region-level VL tasks because predicting bounding boxes is typically hard in end-to-end settings.
On the other hand, MDETR [29] and GLIP [38] are designed to predict bounding boxes, but have not been shown to support tasks such as image captioning and retrieval.
Further, fine-grained pretraining not only requires data with bounding box annotations that are cumbersome to acquire, but the requirement of high input image resolution makes pre-training very costly, especially when using standard Transformer architectures [69] that have quadratic complexity in the size of the image.
A natural but challenging question arises: can we have a unified framework for efficient VL pre-training that benefits both image-level and region-level VL tasks (e.g., both VQA and OD)?
We answer this question by proposing two ideas: ( i ) a novel model architecture that can handle various types of tasks and pre-training strategies (high and low resolution inputs, image and region level outputs) more efficiently than previous work (see Section 3.1 and 4), and ( ii ) a two-stage pre-training pipeline.
In terms of architecture , we present F IBER , shown in Figure 2, which performs deep multimodal fusion in the backbone.
Specifically, instead of having a few dedicated transformer layers on top of the image and text encoders for fusion ( e.g. , as is commonly done in previous work [40, 8, 16, 29, 38]), we propose to directly insert cross-attention modules into the image and text backbones.
Additionally, we support the ability to switch between a dual encoder (for fast image retrieval) and a fusion encoder (for VQA and captioning) readily, by switching on or off the cross-attention modules.
With the same model architecture, by simply adding an object detection
FFN
Self-Att
Cross-Att
FFN
Self-Att
Cross-Att
FFN Self-Att FFN Self-Att
Vision Input
Language Input (a) VQA/Visual Reasoning
FFN
Self-Att
FFN
Self-Att
FFN Self-Att FFN Self-Att
Vision Input
Language Input (b) Retrieval
FFN
Self-Att
FFN Self-Att
Vision Input
Language Input
FFN
Self-Att
Cross-Att
FFN Self-Att (c) Captioning
FFN
Self-Att
Cross-Att
FFN
Self-Att
Cross-Att
FFN Self-Att FFN Self-Att
Vision Input
[Figure]
head ( e.g. , Dynamic Head [12]) on top, F IBER can be readily extended to visual grounding, referring expression comprehension and (open-vocabulary) OD tasks as well.
By
considering
the
nature of different VL tasks, F IBER is pre-trained with a coarse-to-fine two-stage pipeline, as detailed in Figure 1.
Specifically,
• During coarse -grained pre-training, F IBER takes low-resolution (384 × 384) images as input
and is pre-trained with image-text matching, masked language modeling, and image-text contrastive losses, as used in previous work [16, 74, 72].
The pre-trained model can then be directly finetuned for VQA and image captioning tasks (Figure
3a
and 3c
By switching off the cross-attention modules, F IBER also automatically functions as a dual encoder for fast image-text retrieval (Figure 3b
• During fine -grained pre-training, F IBER uses the coarse pre-trained model as initialization, in addition to randomly initialized parameters for the OD head.
At this stage, the model takes highresolution (
800 × 1,333) images as input, and is pre-trained with bounding box localization loss and word-region alignment loss, as used in GLIP [38].
We use image-text-box data with ground-truth box annotations for pre-training, and the model can be directly fine-tuned for grounding and detection tasks (Figure 3d).
Compared to fine-grained pre-training, coarse-grained pre-training is easier to scale up, as it only requires paired image-text data which can be easily harvested from the web.
Crucially, we show that re-using all the parameters from our coarse-grained pre-trained model for fine-grained pre-training alleviates the requirement for large amounts of box-level annotated data.
In our experiments, we show that on fine-grained tasks such as Flickr30k Entities, F IBER using coarse-grained pre-training achieves gains even over previous SoTA (GLIP [38]) that uses 25 × more box-level annotated images during the fine-grained pre-training stage.
We also show that our architecture is much more efficient in terms of training time on OD tasks, as compared to GLIP .
F IBER is the first end-to-end VLP model that can support VL tasks encompassing image-level and region-level outputs.
We conduct experiments on VQAv2 [4], NLVR 2 [66], COCO captioning [45],
NoCaps [1], COCO and Flickr30k image-text retrieval [54], as well as on phrase grounding [54], referring expression comprehension [83], COCO and LVIS detection [20], and a suite of 13 object detection in the wild datasets [38].
We show that our model can provide consistent performance improvement over strong baselines ( e.g. , METER [16] and GLIP [38]) across tasks.
Related Work
VLP for Classical VL Tasks.
ViLBERT [49] and LXMERT [68] were the first two methods to introduce using transformers for VLP.
Since then, we have witnessed a boom of VLP methods [37, 34, 65, 81, 25, 79, 89, 42, 9, 39].
Early methods mainly focus on the use of pre-trained object detectors to extract image region features offline, such as UNITER [8], OSCAR [40], VILLA [18] and VinVL [87].
More recently, end-to-end VLP methods that use the image directly as input have become popular.
In these approaches, convolution networks or vision transformers [15] are used as the image backbone, with additional transformer layers for modeling multimodal fusion [27, 26, 32, 76, 36, 72].
Prominent examples along this line include ViLT [32], ALBEF [36], SimVLM
[75], METER [16], X-VLM [85] and BLIP [35].
These models have achieved the current SoTA on major VL benchmarks such as
VQA and image captioning.
However, they cannot be directly used for tasks such as object detection.
Model
[Table]
Captioning
[Table]
[Table]
✓ ✕ ✕ ✓ ✕ ✕
[Table]
[Table]
✓ ✕ ✓ ✓ ✕ ✕
[Table]
[Table] [Table]
] ✓ ✕ ✕ ✕ ✕ ✓
[Table]
[Table]
[Table]
✕ ✓ ✕ ✕ ✕ ✓
[Table]
] ✓
[Table]
✓ ✕
[Table]
✕
[Table]
✓ ✕ ✓ ✕ ✕ ✓
[Table]
✓ ✕
[Table]
✕ ✓
[Table]
✓ ✓ ✕
[Table]
✓ ✓ ✓
[Table]
✓ ✓ ✓ ✓ ✓ ✓
VLP for Vision Tasks.
Recently, it has been shown that image-text data can be used to learn image encoders from scratch [13, 59].
By performing large-scale contrastive pre-training, CLIP [55] and
ALIGN [28] display strong zero-shot image classification capabilities.
While these models mainly tackle image-level understanding tasks, MDETR [29] extends the end-to-end OD model DETR [6], and uses contrastive learning along with an alignment loss to learn correspondences between image regions and text phrases, opening up the possibility to tackle tasks such as phrase grounding and long-tailed OD using VL models.
This has inspired many follow-up works to further enhance the pre-training [41, 80, 50, 77], among which GLIP [38] shows that OD can also be cast as a VL task ( i.e. , phrase grounding).
However, it has not been shown how traditional VL tasks such as VQA, captioning and retrieval can be well supported in GLIP [38] and MDETR [29].
Unified VL Modeling.
There have been a few recent attempts that try to develop unified VL models.
VL-T5 [9] unifies VL tasks as text generation; however, pre-trained object detectors are used for image feature extraction, so the model cannot be end-to-end pre-trained.
UniT [23] proposes a multimodal multi-task framework with a unified transformer; however, it can only support VQA and object detection tasks, but not captioning and grounding.
GPV [21] proposes a general-purpose vision system, and FLAVA [64] presents a VL system similar to METER [16]; however, they did not evaluate on grounding and detection tasks, and their performance on other VL tasks is still far from SoTA.
UniTAB [78] and OFA [73] reformulate grounding as a sequence generation task, by borrowing ideas from Pix2Seq [7].
However, these approaches have not been demonstrated to work on standard OD benchmarks, and also cannot be used as dual encoders for fast image retrieval.
Our model is the first work that can support not only VQA, image captioning and O ( n + m ) retrieval, but also visual grounding and object detection, with impressive performance across all tasks.
A detailed comparison is provided in Table 1.
Method
In this section, we first describe the proposed model architecture in Section 3.1.
We then illustrate our two-stage pre-training paradigm in Section 3.2, followed by fine-tuning strategies for all the tasks supported by F IBER in Section 3.3.
3.1 Fusion in the Backbone
[Figure]
The architecture of F IBER is shown in Figure 2.
Different from models that stack a modality fusion module on top of the vision or language backbones [8, 16], we insert multimodal fusion inside the backbones, and include a gating mechanism for the cross-modal layers (shown in
Figure 4).
Specifically, at each encoding layer, we have:
[Equation]
where α is a learnable parameter initialized to 0.
For simplicity, we insert the same number of cross-attention layers into the vision and language backbones.
By inserting cross-attention layers with the gating mechanism, we enable cross-modal interactions without affecting the original computational flow of the backbones at the beginning of model training.
Also, we can easily switch off the interactions by setting α to 0, and the backbones can be used in the dual-encoder setting.
In addition, compared to stacking a large number of transformer layers on top of the backbones, our approach of inserting cross-attention layers is relatively light-weight and thus more memory-efficient.
To illustrate, both GLIP [38] and METER [16] use an additional 110M modality fusion parameters for a base-size model, while F IBER only adds about 26M parameters.
During training, the fusion module of F IBER only consumes half of the FLOPs needed by METER (12.35 vs. 24.04 GFLOPs for one instance).
We experimented with two other model variants for fusion in the backbone, the details of which are provided in Appendix.
3.2 Coarse-to-Fine Pre-training
We divide VL tasks into two categories based on whether or not we need to generate region-level outputs on the image side.
While these two kinds of tasks are characteristically different, they both require fusion between the vision and language modalities, and we hypothesize that sharing as many parameters as possible between the model used for these two sets of tasks will be beneficial.
Based on this motivation, we propose a two-stage pre-training paradigm, where we first pre-train models with image-level objectives on images at low resolution, and then perform further pre-training with region-level objectives where the input images are at a higher resolution.
In this way, the coarsegrained supervision from the first stage can provide good initialization for the second stage for all the shared parameters.
F IBER with the same architecture (Swin Transformer [47] and RoBERTa [46]) is used as the backbone for both stages of pre-training.
Coarse-grained Pre
-training.
For tasks like VQA and captioning, it has been demonstrated [36, 16, 74] that masked language modeling (MLM), image-text matching (ITM), and image-text contrastive (ITC) objectives are helpful for ViT-based VLP models.
Following previous work, we use all the three objectives during pre-training.
Specifically,
• For ITC , the inserted cross-attention modules are switched off, so F IBER functions as a dual encoder.
Given a batch of N image-caption pairs, we first compute their representations with our vision and language encoders independently without modality fusion, and then maximize the similarities between N positive image-text pairs while minimizing the similarities between the rest N 2 − N negative pairs, via a contrastive loss.
• For MLM and ITM , the inserted cross-attention modules are switched on, so F IBER now functions as a fusion encoder.
For MLM, we randomly mask 15% of the input tokens and the model is trained to reconstruct the original tokens.
For image-text matching, the model is given an image-text pair and predicts whether they are matched.
Following previous work [36, 74], we sample hard negatives based on the similarities computed from the above ITC loss.
Fine-grained Pre-training.
Most existing VL architectures [8, 68, 29, 35, 73, 9] use vanilla transformers both for encoding the vision as well as language inputs.
However, in contrast to tokens in text, the entities of interest in images do not all occur at the same scale.
Being able to accurately model the image at different scales is especially important for tasks such as object detection and phrase grounding.
To handle this, it is typical in object detection literature to use input images at higher resolutions (800 × 1333), which becomes problematic when using vanilla transformers that scale quadratically in the length of the input sequence.
As mentioned earlier, we use a Swin Transformer [47] as our image encoder, which provides hierarchical representations of the image while having linear complexity in the size of the image.
We combine these multi-scale representations using an
FPN [43] for object detection training.
For fine-grained pre-training, we switch on the cross-attention modules, using F IBER as a fusion encoder.
This ensures that the image representations that are passed to the FPN are already text-aware, and is a crucial difference compared to GLIP [38], where the image-text fusion takes place in the object detection head.
Once the text-aware image features are extracted by the Swin backbone and image-aware text features are extracted using RoBERTa [46], the image features after the FPN are fed to a DynamicHead [12] which predicts a set of regions.
Just as in [38], we compute the dot product between the image region features R TA and the contextualized token representations T IA to compute the grounding score:
[Equation]
where R TA represents regions that are text aware, produced using the OD-Head that takes as input
I TA , which are image representations that are already text-aware and T IA are the text features that have already attended to the image features.
The typical object detection model has a classification head that predicts the label of the object, and a localization head that predicts the bounding box.
We follow GLIP [38] by substituting the classification head with the grounding score S GROUNDING .
The localization loss is composed of two parts: a centerness loss and GIoU loss, which are used to supervise the box prediction.
Taken together, F IBER learns the correspondence between regions in the image and phrases in the text, making it possible to tackle tasks such as phrase grounding and object detection using the same framework.
We use ATSS framework [88] in our paper, but our method can be combined easily with other object detectors such as Faster-RCNN [57] and RetinaNet [44] as well.
3.3 Adaptation to Downstream Tasks
We now describe how we adapt F IBER to different downstream tasks as depicted in Figure 3.
• For VL classification tasks such as VQA , we use F IBER as a fusion encoder.
Specifically, the top M layers of the vision and language backbones interact with each other and produce multimodal representations.
The final layer representations of the two modalities are concatenated together to generate the final outputs for tasks such as VQA and visual reasoning.
• For retrieval tasks , we switch off the inserted cross-attention modules to use F IBER as a dual encoder for fast image-text retrieval.
• For captioning , we adapt F IBER by only keeping the image-to-text cross-attentions and using causal masks in the decoding side.
The representations of the final image encoding layer are fed into the cross-attention modules.
In this way, the model is turned into a seq2seq model [67, 10] and performs captioning in an auto-regressive way.
• For phrase grounding, object detection and referring expression comprehension , we use
F IBER as a fusion encoder, and the OD-Head introduced during fine-grained pre-training receives image features that are already language aware due to the multimodal representations extracted by
F IBER .
The pre-trained model is directly used without any modifications for these tasks.
Experiments
[Table]
Pre-training Datasets.
Following previous work [8, 32, 36, 16, 72, 74], we perform coarsegrained pre-training on COCO [45], Conceptual Captions [62], SBU Captions [51], and Visual Genome [33].
The four datasets consist of about 4M images in total.
For fine-grained pretraining, we use two data sources: data curated by MDETR [29] after removing the COCO images, and the Objects365 [61] detection dataset, together consisting of about 0.8M images.
We ensure that we exclude any data that exists in the validation or test splits of downstream tasks.
Architecture.
We adopt Swin-Base [47] and
RoBERTa-Base [46] as our vision and text backbones, which are initialized with weights from uni-modal pre-training.
We insert cross-attention blocks into the top 6 blocks of the vision and text encoders.
The input resolution is 384 × 384 for coarse-grained pre-training and 800 × 1 , 333 for fine-grained pre-training.
Using a hierarchical vision transformer enables us to efficiently tackle these high resolution tasks, which would be expensive in models such as BLIP [35] that rely on the vanilla transformer architecture.
In METER [16], which does explore using a Swin transformer as the image encoder, the multi-modal fusion occurs in layers specifically designed to align the modalities, only after the image and text features are extracted from the uni-modal backbones.
This is in contrast to our approach where the hierarchical image features that are used in the FPN for fine-grained training
[Table]
are already language aware, due to the multi-modal fusion being in the backbone.
This also lets us avoid adding additional “language-aware deep fusion layers” [38] as part of the OD head as in
GLIP, resulting in 1.5x faster training while maintaining performance as shown in Table 2.
While in principle it would be possible to use the image features extracted by METER’s backbone for object detection, it would be necessary as in GLIP to add additional layers to make the visual features “language-aware” for good detection performance, especially on datasets with limited training data and with rare and infrequent objects.
Implementation Details.
We perform coarse-grained pre-training for 100k steps with 4,096 batch size on 64 A100 GPUs.
We use AdamW [48] with the peak learning rates of 1e-4 for the backbones and 5e-4 for the cross-modal parameters.
We use linear warmup over the first 1k steps and linear decay.
For fine-grained pre-training, we train for 800k steps on 64 V100 GPUs, with a batch size of 64.
We use a learning rate of 1e-5 for the language backbone, and 1e-4 for the rest of the model with a weight decay of 0.01.
We use a linear warmup over the first 2k steps and then a constant learning rate, with two learning rate drops by a factor of 10 at 67% and 89% of the total number of steps.
4.1 Results on Downstream Tasks
Vision-Language Classification.
We first experiment on two representative VL classification tasks, including VQAv2 [4] and NLVR 2 [66].
As reported in Table 3, we achieve the best performance compared to other models in the same setting.
It is worth noting that F IBER pre-trained with 4M images can achieve better performance than BLIP trained with 129M images and SimVLM trained with 1.8B images.
The results indicate that introducing fusion modules into the backbone is an effective alternative to appending them on the top of uni-modal backbones.
Image-Text Retrieval.
In Table 3 we report image retrieval performance in the dual encoder setting, achieving competitive performance on both Flickr30k [54] and COCO [45] retrieval tasks.
However, previous work has shown that fusion encoders obtain superior performance, albeit at the cost of efficiency as it involves feeding every image-text pair into the model.
To illustrate, on the COCO test data, ranking the similarities between 5K images and 25K captions requires the model to process each image-caption pair 75M times, whereas the dual encoder model only needs 30K forward passes.
As shown in Table 4, the fusion encoder can indeed surpass the dual encoder on retrieval tasks by a large margin.
In addition, directly ensembling the two models by summing their similarity scores together for each image-caption pair can bring us huge improvements.
[Table] [Table]
is
[Table]
re
[Table]
rank
[Table]
also provide a full set of
[Table]
on all metrics
[Table]
Further, we explore combining the strengths of both strategies by performing re-ranking as in [19, 35, 36].
Specifically, we first retrieve the top- k most similar instances using the dual encoder setup
[Table]
and then add the similarity scores between the given instance and the top- k candidates provided by the fusion encoder to the original scores to perform retrieval.
From Table
[Table]
we can see that this strategy provides a balance between efficiency and performance
[Table]
and that just re-ranking the top-10 instances can achieve comparable performance with ensembling.
Image Captioning.
We also evaluate our models on COCO [45] and NoCaps [1] captioning to test whether F IBER can be adapted to generation tasks.
As in Table 5, F IBER can achieve better performance than models trained on the same data with and without CIDEr optimization [58].
We find that integrating GOLD [52] into F IBER can bring significant improvements, outperforming models trained with hundreds of millions of images.
Notably, we establish the absolute state-of-the-art
CIDEr scores on COCO for base-size models.
Considering that F IBER is not pre-trained to perform captioning, the results demonstrate the strong generalization ability of F IBER .
Phrase Grounding.
Our fine-grained pre-training stage incorporates Flickr30k entities grounding data, and we achieve 87.4 on the Recall@1 metric on the test set without any subsequent fine-tuning.
This not only surpasses the current SoTA [38] using a smaller sized model (Swin-B compared to their
Swin-L), but also uses 25x less fine-grained data.
Our F IBER model is able to leverage the image-text coarse-grained pre-training stage better, instead of relying on expensive pseudo-labelling of large
[Table] [Table]
[Table]
web-scale corpus and subsequent high-resolution training on this generated fine-grained data as in [38].
We also compare our approach without using any coarse-grained VL training (image encoder initialized to Swin-B weights from IN22k, and text encoder initialized to pre-trained RoBERTa), and even in this setting, we are able to outperform a similarly sized GLIP model (GLIP-B), proving that our fusion in the backbone is better at capturing fine-grained image-text understanding.
[Figure]
Referring Expression Comprehension (REC).
In contrast to many previous works [8, 18, 49] that tackle the
REC task by re-ranking object proposals provided by an off-the-shelf detector, we follow [29] to directly predict the bounding box for the given referring expression.
Using our proposed two stage pre-training, F IBER achieves better performance than current SoTA [73] that uses a
Large sized model.
Notably, on RefCOCOg
[84], which contains much longer referring expressions than in RefCOCO/RefCOCO+ [31], we observe more than 2 points boost over OFA-L.
On the challenging testB split of both
RefCOCO and RefCOCO+, F IBER outperforms current
SoTA, OFA-L.
Object Detection.
We report F IBER results on two standard object detection benchmarks, COCO [45] and LVIS [20], in zero-shot transfer 1 as well as fine-tuned settings in Table 8.
The LVIS dataset consists of a long-tail of object classes, and is a popular test-bed for evaluating models on their generalization capabilities and robustness to class imbalance.
On the APr metric, which is the Average Precision on
[Table]
rare objects, F IBER outperforms GLIP-L which is a bigger model and also trained with 25 × more fine-grained data.
We also report zero-shot and fine-tuned results on a suite of 13 ODinW (object detection in the wild) datasets, spanning various domains and show consistent performance improvements over previous
SoTA.
Additionally, in Figure 5, we report few-shot results aggregated across these 13 datasets and show better data efficiency over GLIP-B trained with the same fine-grained data.
Ablation Study.
In Appendix A.2 and A.3, we have provided detailed ablations that guided our architecture design, including ablations on fusion strategies, pre-training objectives, architecture for captioning, and additional results on open-ended VQA, and detailed few-shot ODinW results.
Due to the space limit, these ablations and additional results are only provided in the Appendix.
Some important observations are summarized below.
( i ) Co-attention
works similarly to merged attention for fusion in the backbone.
( ii ) Adding a gating parameter in co-attention allows the addition of fusion in more layers, and also gives better performance than merged attention.
( iii ) Adding co-attention in the last 6 layers provides a balance between performance and efficiency.
( iv ) MLM, ITM with hard negative mining, and ITC are all important pre-training objectives for training FIBER-style models.
Conclusion
We propose ( i ) F IBER , a novel architecture and ( ii ) a coarse-to-fine pre-training pipeline.
We perform extensive experiments and show consistent improvements over strong baselines across a diverse set of tasks.
The results demonstrate the effectiveness of F IBER coupled with our pre-training strategy, by setting new SoTA scores while at the same time reducing the requirement of expensive box-level annotations.
Future directions include scaling our models and extending our framework to other modalities.
The approach introduced in our work can potentially inherit undesirable societal biases that exist in our pre-training data.
Careful debiasing and filtering of data should be undertaken before real-life deployment of our work.
Additionally, pre-training can induce environmental costs, and minimizing these costs is an avenue that we plan to explore further.
Acknowledgement
We would like to thank Nguyen Bach, Jiayuan Huang, and Luis Vargas for their support.
We also thank Wenhui Wang, Li Dong, Furu Wei, Bin Xiao, and Lu Yuan for their helpful discussions.
We also thank Liunian Harold Li and Te-Lin Wu for their feedback on the manuscript.
Aishwarya is supported in part by the National Science Foundation under NSF Award 1922658.
Zi-Yi is supported in part by the DARPA Machine Common Sense (MCS) program under Cooperative Agreement
N66001-19-2-4032 and NIH R01HL152270.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] See Section(c)
Did you discuss any potential negative societal impacts of your work?
[Yes] See Section(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[N/A] (b) Did you include complete proofs of all theoretical results?
[N/A] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] (b) Did you specify all the training details (e
g., data splits, hyperparameters, how they were chosen)?
[Yes] (c) Did you report error bars (e
g., with respect to the random seed after running experiments multiple times)?
[No] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] (b) Did you mention the license of the assets?
[Yes] (c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether
and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
A Appendix A.1 Implementation Details
Vision-Language Classification.
For the VL classification tasks, we follow METER [16] to set the hyper-parameters.
Specifically, we fine-tune our models with the peak learning rates of 2e-5 for the backbones, 1e-4 for the cross-modal parameters, and 1e-3 for the head layer for 10 epochs.
The batch size is set to 512.
The image resolutions are set to 576 for VQAv2 and 384 for NLVR 2 .
We evaluate models with the VQA scores for VQAv2 and accuracy for NLVR 2 .
RandAugment [11] is used during the downstream fine-tuning stage.
Image-Text Retrieval.
For image-text retrieval, we remove the cross-attention layers in the backbones and use the dual encoder architecture.
We set the peak learning rates to 2e-5 for the backbones and 1e-4 for the head layer.
The batch size is set to 1024.
The image resolutions are set tofor both COCO and Flickr30k.
We evaluate on the Recall@1,5,10 metrics for both text and image retrieval.
Image Captioning.
For image captioning, we only keep the image-to-text attentions and feed the image representations in the last layer of the image encoder to the cross-attention modules.
In this way, the model is turned into a standard seq2seq model, and we use the causal mask in the decoding side and predict outputs auto-regressively.
We first train our models with the cross-entropy loss for 5 epochs with the peak learning rates of 5e-5 for the backbones, and 2.5e-4 for the rest of the parameters.
Then, we fine-tune it with GOLD [52] for 5 epochs as it is efficient and has proven to be effective when the model input can correspond to different outputs.
We set the peak learning rate to 1e-5 for the backbones during GOLD training.
For CIDEr optimization, the learning rate is further reduced to 1e-6 and we train the models for 3 epochs.
The batch size is set to 512.
We use a beam size of 5 during inference and do not use constrained beam search.
We use the same model when testing on COCO and NoCaps, and we evaluate on BLEU [53], METEOR [5], CIDEr [70], and
SPICE [3] metrics.
Phrase Grounding.
For phrase grounding on Flickr30k, we do not further fine-tune the model after fine-grained pre-training, and just directly evaluate on the Recall@ 1,5,10 metrics.
Referring Expression Comprehension (REC).
For the REC datasets, we use batch size 16 and fine-tune on the respective dataset for 20 epochs.
We use a warmup of 2000 steps, with a peak learning rate of 1e-5 for both the OD head as well as the rest of the model’s paramaters, with two learning rate drops at 67% and 89% of the total number of steps.
We switch off the horizontal flip augmentation during REC training, as we find that it adversely affects the performance, especially on the RefCOCO dataset, which includes many examples having degenerate language such as just “left” or “right” rather than using descriptive words for the referring expressions.
Object Detection.
For both COCO and LVIS detection, we train for 24 epochs, with batch size 32, with a learning rate of 1e-5 for the whole model, with two learning rate drops at 67% and 89% of the total number of steps.
For the ODinW datasets, we fine-tune for 12 epochs, with early stopping based on the validation accuracy.
The object detection data is constructed as follows - The object category names are directly used in their text form separated by full stops as input to the text encoder.
We follow the same protocol as in
GLIP [3] to be comparable to their experiments.
More specifically, the input text will look like this: "person.
bicycle. car.
.... toothbrush", and the model will learn how to ground image regions into these object names.
An example of input and output predicted by the model can be seen in Fig.
A.2 Ablation Study
Ablation Study on the Fusion Strategies.
We perform ablation studies on our fusion module.
We investigate three different fusion strategies as shown in Figure 6.
Merged attention concatenates representations from the two input modalities and feeds them into the self-attention layer for fusion.
Note that here the key and value matrices for the two modalities are different.
On the other hand, co-attention inserts a cross-attention layer into each of the encoding layer.
The insertion of the
Self-Attention
Feedforward x
K x/y V x/y y
Q x (a) merged attention
Self-Attention
Cross-Attention
Feedforward
Q x y x
K x V x
Q x K y
V y (b) co-attention w/o α
Feedforward α
Self-Attention
Cross-Attention
Q x y x
K x V x
Q x K y
V y (c) co-attention w/
α
Model
[Table] [Table]
stage.
cross-attention layer offers the flexibility of controlling to what extent we want the two modalities to fuse together as we can easily introduce an α term into the module as in Figure 4.
As shown in Table 9, we compare the three fusion strategies by directly fine-tuning our models without performing VLP for efficiency.
We use Swin Transformer and RoBERTa as our vision and text backbones and load their pre-trained parameters for initialization.
We set the image resolution to 224 × 224.
We can see that merged attention and co-attention achieve comparable performance without α .
For both strategies, increasing the number of fusion layers can lead to performance drop.
However, after introducing α , we can see significant improvements of co-attention, indicating the importance of having an explicit controlling/gating mechanism for fusion in the backbone.
After the α term is introduced, we can increase the number of fusion layers and achieve robust performance.
Based on the ablation results, we choose to fuse the top 6 layers of the backbones as it can achieve a good accuracy-efficiency trade-off.
Ablation Study on Pre-training Objectives.
Following previous work [36, 16, 74], we pre-train our models with image conditioned masked language modeling, image-text matching with hard negative mining, and image-text contrastive losses during the coarse-grained pre-training stage.
In
[Table] [Table]
this part, we ablate each of the pre-training objectives and evaluate our models on both VQAv2 and
Flickr30k retrieval tasks.
Specifically, we use Swin Transformer and RoBERTa as our vision and text backbones and load their pre-trained parameters for initialization.
The image resolution is set to 224 × 224 and we pre-train models for 100k steps with 1,024 batch size.
We use AdamW with the peak learning rates of 1e-4 for the backbones and 5e-4 for the cross-modal parameters.
We use linear warmup over the first 1k steps and linear decay.
As shown in Table
[Table]
we can see that removing any of the pre-training objectives can lead to performance drop, and hard negative mining can bring improvements on both VQA and retrieval tasks.
Masked language modeling is most effective for VQA, while removing it will not hurt the retrieval performance.
This set of experiments demonstrates that all of the objectives are necessary for our models to obtain good performance
[Table]
Ablation Study on the Two-Stage Pre-training.
In this paper, we propose a coarse-to-fine pretraining strategy for handling VL tasks of different kinds.
In this paragraph, we remove the coarsegrained pre-training stage and only pre-train the models with image-text-box data and see how it performs.
As shown in Table 11, we see gains across both tasks when utilizing the coarse-grained pre-training.
Similar to the case of Flickr30k, on RefCOCO+ the coarse-grained pre-training helps
FIBER to get better performance than large-sized model trained with more data.
In addition, note that without the coarse-grained pre-training, the only difference between FIBER and GLIP is the architectural difference, and the fact that FIBER can still outperform GLIP in this setting demonstrates the effectiveness of our proposed architecture.
Ablation Study on Different Backbones.
While previous work [16] has compared different vision and text backbones for VLP models, we investigate if their conclusions stil apply in our settings.
Specifically, we try BERT and RoBERTa for our text encoder and CLIP-ViT and Swin Transformer for our image encoder.
As shown in Table 12, we can see that RoBERTa
and Swin Transformer perform slightly better than BERT and CLIP-ViT before VLP, which is consistent with previous findings in
METER [16].
Note that while CLIP-ViT has the potential to perform better than Swin Transformer after VLP, it is hard to be adapted for region-level tasks such as object detection.
Therefore, pairing
Swin Transformer with RoBERTa is the optimal configuration in our settings.
A.3 Additional Results
Additional Results on Image Captioning.
For image captioning, we evaluate the models with
BLEU-4, METEOR, ROUGE-L, CIDEr, and SPICE metrics on COCO and NoCaps.
On NoCaps, we have fine-grained evaluation results on different domains, including in-domain, near-domain, out-domain, and entire domain settings.
In this part, we provide the complete evaluation results in
Table 13, 14 and 15.
We can see that both GOLD and CIDEr optimization can improve the model performance across metrics.
We also see a noticeable performance drop when evaluating our models on out-of-domain data, but complementary methods such as constrained beam search can be used to
FFN
Self-Att
FFN
Self-Att
Vision Input
Language Input
FFN
Self-Att
Cross-Att
FFN
Self-Att (a) seq2seq
FFN
Self-Att
FFN
Self-Att
Vision Input
Language Input
FFN
Self-Att
Cross-Att
FFN
Self-Att (b) ladder
Model
[Table]
alleviate the issue.
Also, training our models with more captioning data should also be helpful in these settings.
Also
[Table]
in the main paper
[Table]
we adapt F IBER for image captioning by turning it into a standard seq2seq model as in Figure 7a, where the output of the final encoding layer will be fed into the image-to-text cross-attention modules.
Another possible design is to keep the ladder structure as we used in pre-training (Figure 7b), so there can be less mismatching between pre-training and fine-tuning.
As shown in Table 13, the two architectures can achieve comparable performance.
Considering that the seq2seq architecture is more widely adopted in the current literature, we decide to use the seq2seq architecture for image captioning.
[Table]
Open-ended VQA.
In most existing literature,
VQA is treated as a classification task, where a vocabulary of some most frequent answers are constructed and VL models predict which answer corresponds to the given question based the constructed vocabulary.
However, question answering is inherently open-ended.
Since we can turn our models into a generative model by fine-tuning on image captioning, we also investigate if our models can perform open-ended
VQA in this part.
Following [9], we break down VQA questions into in-domain and out-of-domain questions,
[Table]
[Table] [Table]
where the answers to the out-of-domain questions do not appear in the top- k ( k = 3 , 129 ) candidates.
We use the Karpathy split [30] in this setting.
As shown in Table 16, our generative model can perform better than VL-T5 and VL-BART, while lagging behind SimVLM especially in out-of-domain settings, possibly because SimVLM is trained with over a billion image-caption pairs and is more robust in this setting.
The results indicate that our model can be turned into a general open-ended VQA model as well.
Uni-modal Performance.
It can be interesting to see whether our backbones can still perform uni-modal tasks after VLP.
Therefore, in this part, we also evaluate our language backbones on uni-modal tasks.
[Table]
Specifically, we test our language backbone after the first coarse-grained pre-training stage on the
GLUE [71] benchmark.
As shown in Table 17, the uni-modal performance of our text encoder can drop marginally on some tasks, possibly because the model only encounters simple text captions during VLP.
However, it is still better than SimVLM which is trained with 800GB of web-crawled documents from scratch.
On the other hand, as shown in Table 18, our image encoder can achieve comparable and sometimes even better performance on vision-only tasks including image classification and semantic segmenta-
[Table]
tion after coarse-grained pre-training.
The results suggest that the image encoder can remain powerful even if the fusion modules are removed.
Using the Model Checkpoint After Fine-grained Pre-training for VQA.
We also test what if we fine-tune the fine-grained pre-trained checkpoint on VQA in this part.
We find that after the second-stage fine-grained pre-training, the model performance on the VQAv2 test-dev set can drop from 78.55 to 74.3, indicating that the two-stage pre-training paradigm is indeed necessary for different VL tasks, as tasks of different characteristics can require checkpoints from different pre-training stages.
Detailed Results on ODinW. Detailed results on the 13 ODinW datasets are provided in Table 19.
A.4 Visualization after coarse-grained pre-training
We also provide a qualitative analysis of our model.
As shown in Figure 8, we use Grad-CAM [60] to visualize the cross-attention maps of our coarse-grained pre-trained checkpoint.
We find that the model can correctly align concepts and image regions for some examples, suggesting that the model can learn visual grounding implicitly.
A.5 Visualization after fine-grained pre-training
We also provide visualization after fine-grained pre-training in Figure
A.6 Text and Vision Backbones of Different Models
In this section, we list the backbones of different models in Table 20.
[Table]