Coarse-to-Fine Vision-Language Pre-training with Fusion in the Backbone
Anonymous Author(s)
AffiliationAddress email
Abstract
Vision-language (VL) pre-training has recently received considerable attention.
However, most existing end-to-end pre-training approaches either only aim totackle VL tasks such as image-text retrieval, visual question answering (VQA)and image captioning that test high-level understanding of images, or only targetregion-level understanding for tasks such as phrase grounding and object detection.
We present F IBER ( F usion- I n-the- B ackbone-based transform ER ), a new VL modelarchitecture that can seamlessly handle both these types of tasks.
Instead of havingdedicated transformer layers for fusion after the uni-modal backbones, F IBERpushes multimodal fusion deep into the model by inserting cross-attention into theimage and text backbones, bringing gains in terms of memory and performance.
Inaddition, unlike previous work that is either only pre-trained on image-text data oron fine-grained data with box-level annotations, we present a two-stage pre-trainingstrategy that uses both these kinds of data efficiently: ( i ) coarse -grained pre-trainingbased on image-text data; followed by ( ii ) fine -grained pre-training based on imagetext-box data.
We conduct comprehensive experiments on a wide range of VL tasks,ranging from VQA, image captioning, and retrieval, to phrase grounding, referringexpression comprehension, and object detection.
Using deep multimodal fusioncoupled with the two-stage pre-training, F IBER provides consistent performanceimprovements over strong baselines across all tasks, often outperforming methodsusing magnitudes more data.
Code will be released upon acceptance.
Introduction
Inspired by the success of language model pre-training [11, 50, 41], coupled with the unification ofarchitectures used in the NLP and computer vision communities [12, 4], vision-language pre-training(VLP) [61, 44, 33, 6] has been receiving an increasing amount of attention.
It has been proven that
VLP can establish state-of-the-art performance on visual question answering [3], visual reasoning [59],image captioning, and image-text retrieval [40].
The pre-training objectives commonly used for thesetasks, such as image-text matching, image conditioned masked language modeling and image-textconstrastive learning, require multimodal understanding at the image level.
Typically, this means thepre-training is done using images at lower resolution ( e.g. , 384 × 384), making it possible to scale uptraining by using large batch sizes.
Recently, it has also been shown that tasks such as image classification and object detection (OD),which have been traditionally viewed as vision-only tasks, can benefit from being cast as VLtasks [49, 24, 34, 25].
Inspired by MDETR [25], GLIP [34] reformulates standard classificationbased OD as phrase grounding.
This opens up the possibility to leverage VLP for OD, and vice versa,and this unification has led to impressive performance on several established OD as well as phrasegrounding benchmarks [48].
Since these tasks involve fine-grained image understanding betweenregions in the image and phrases in the text, and also require prediction of precise bounding boxes atthe output, the pre-training typically involves using high resolution input images ( e.g. , 800 × 1,333).
Existing multimodal architectures typically do not support both kinds of tasks.
Specifically, the fullyend-to-end VLP models such as ALBEF [32], METER [13], and SimVLM [66] can achieve thestate of the art (SoTA) on image-level understanding tasks, but it is non-trivial to extend them forregion-level VL tasks because predicting bounding boxes is typically hard in end-to-end settings.
On the other hand, MDETR [25] and GLIP [34] are designed to predict bounding boxes, but havenot been shown to support tasks such as image captioning and retrieval.
Further, fine-grained pretraining not only requires data with bounding box annotations that are cumbersome to acquire, butthe requirement of high input image resolution makes pre-training very costly, especially when usingstandard Transformer architectures [62] that have quadratic complexity in the size of the image.
Anatural but challenging question arises: can we have a unified framework for efficient VL pre-trainingthat benefits both image-level and region-level VL tasks (e.g., both VQA and OD)?
We answer this question by proposing two ideas:(
i ) a novel model architecture that can handlevarious types of tasks and pre-training strategies (high and low resolution inputs, image andregion level outputs) more efficiently than previous work (see Section 3.1 and 4), and ( ii ) atwo-stage pre-training pipeline.
In terms of architecture , we present F IBER ,shown in Figure 2, which performs deep multimodal fusion in the backbone.
Specifically,instead of having a few dedicated transformerlayers on top of the image and text encoders forfusion ( e.g. , as is commonly done in previouswork [36, 6, 13, 25, 34]), we propose to directlyinsert cross-attention modules into the imageand text backbones.
Additionally, we supportthe ability to switch between a dual encoder (forfast image retrieval) and a fusion encoder (for VQA and captioning) readily, by switching on or offthe cross-attention modules.
With the same model architecture, by simply adding an object detectionhead ( e.g. , Dynamic Head [9]) on top, F IBER can be readily extended to visual grounding, referringexpression comprehension and (open-vocabulary) OD tasks as well.
By considering the nature of different VL tasks, F IBER is pre-trained with a coarse-to-fine two-stagepipeline, as detailed in Figure 1.
Specifically,
FFN
Self-Att
Cross-Att
FFN
Self-Att
Cross-Att
FFN Self-Att FFN Self-Att
Vision Input
Language Input (a) VQA/Visual Reasoning
FFN
Self-Att
FFN
Self-Att
FFN Self-Att FFN Self-Att
Vision Input
Language Input (b) Retrieval
FFN
Self-Att
FFN Self-Att
Vision Input
Language Input
FFN
Self-Att
Cross-Att
FFN Self-Att (c) Captioning
FFN
Self-Att
Cross-Att
FFN
Self-Att
Cross-Att
FFN Self-Att FFN Self-Att
Vision Input
[Figure]
Input
[Figure]
• During coarse -grained pre-training, F IBER takes low-resolution (384 × 384) images as input, and ispre-trained with image-text matching, masked language modeling, and image-text contrastive losses,as used in previous work [13, 65, 63].
The pre-trained model can then be directly finetuned forVQA and image captioning tasks (Figure 3a and 3c).
By switching off the cross-attention modules,F IBER also automatically functions as a dual encoder for fast image-text retrieval (Figure 3b).
• During fine -grained pre-training, F IBER uses the coarse pre-trained model as initialization, inaddition to randomly initialized parameters for the OD head.
At this stage, the model takes highresolution (800 × 1,333) images as input, and is pre-trained with bounding box localization loss andword-region alignment loss, as used in GLIP [34].
We use image-text-box data with ground-truthbox annotations for pre-training, and the model can be directly fine-tuned for grounding and
detection tasks (Figure 3d).
Compared to fine-grained pre-training, coarse-grained pre-training is easier to scale up, as it onlyrequires paired image-text data which can be easily harvested from the web.
Crucially, we show thatre-using all the parameters from our coarse-grained pre-trained model for fine-grained pre-trainingalleviates the requirement for large amounts of box-level annotated data.
In our experiments, weshow that on fine-grained tasks such as Flickr30k Entities, F IBER using coarse-grained pre-trainingachieves gains even over previous SoTA (GLIP [34]) that uses 25 × more box-level annotated imagesduring the fine-grained pre-training stage.
We also show that our architecture is much more efficientin terms of training time on OD tasks, as compared to GLIP .
F IBER is the first end-to-end VLP model that can support VL tasks encompassing image-level andregion-level outputs.
We conduct experiments on VQAv2 [3], NLVR 2 [59], COCO captioning [40],
NoCaps [1], COCO and Flickr30k image-text retrieval [48], as well as on phrase grounding [48],referring expression comprehension [74], COCO and LVIS detection [16], and a suite of 13 objectdetection in the wild datasets [34].
We show that our model can provide consistent performanceimprovement over strong baselines ( e.g. , METER [13] and GLIP [34]) across tasks.
Related Work
VLP for Classical VL Tasks.
ViLBERT [44] and LXMERT [61] were the first two methods tointroduce using transformers for VLP.
Since then, we have witnessed a boom of VLP methods [33, 30,58, 72, 21, 70, 80, 29, 7, 35].
Early methods mainly focus on the use of pre-trained object detectors toextract image region features offline, such as UNITER [6], OSCAR [36], VILLA [15] and VinVL [78].
More recently, end-to-end VLP methods that use the image directly as input have become popular.
Inthese approaches, convolution networks or vision transformers [12] are used as the image backbone,with additional transformer layers for modeling multimodal fusion [23, 22, 27, 67, 32, 63].
Prominentexamples along this line include ViLT [27], ALBEF [32], SimVLM [66], METER [13], X-VLM [76]and BLIP [31].
These models have achieved the current SoTA on major VL benchmarks such as VQA and image captioning.
However, they cannot be directly used for tasks such as object detection.
VLP for Vision Tasks.
Recently, it has been shown that image-text data can be used to learn imageencoders from scratch [10, 53].
By performing large-scale contrastive pre-training, CLIP [49] and
ALIGN [24] display strong zero-shot image classification capabilities.
While these models mainlytackle image-level understanding tasks, MDETR [25] extends the end-to-end OD model DETR [4],and uses contrastive learning along with an alignment loss to learn correspondences between image
Model
VQA † O ( n + m ) Retrieval ‡
Captioning Grounding OD End2End
ViLBERT [44], LXMERT [61], UNITER [6] ✓ ✕ ✕ ✓ ✕ ✕
OSCAR [36], VinVL [78] ✓ ✕ ✓ ✓ ✕ ✕
PixelBERT [23], CLIP-ViL [56], ViLT [27] ✓ ✕ ✕ ✕ ✕ ✓
CLIP [49] ∗ , ALIGN [24] ✕ ✓ ✕ ✕ ✕ ✓
VL-T5 [7] ✓ ✕ ✓ ✕ ✕ ✕
METER [13], SimVLM [66] ✓ ✕ ✓ ✕ ✕ ✓
ALBEF [32], FLAVA [57], VLMo [65] ✓ ✓ ✕ ✕ ✕ ✓
BLIP [31], CoCa [73], Flamingo [2] ✓ ✓ ✓ ✕ ✕ ✓
MDETR [25], GLIP [34] ✓ ✕ ✕ ✓ ✓ ✓
UNICORN [69], X-VLM [76], OFA [64] ✓ ✕ ✓ ✓ ✕ ✓
F IBER ✓ ✓ ✓ ✓ ✓ ✓ regions and text phrases, opening up the possibility to tackle tasks such as phrase grounding andlong-tailed OD using VL models.
This has inspired many follow-up works to further enhance thepre-training [37, 71, 45, 68], among which GLIP [34] shows that OD can also be cast as a VL task( i.e. , phrase grounding).
However, it has not been shown how traditional VL tasks such as VQA,captioning and retrieval can be well supported in GLIP [34] and MDETR [25].
Unified VL Modeling.
There have been a few recent attempts that try to develop unified VL models.
VL-T5 [7] unifies VL tasks as text generation; however, pre-trained object detectors are used forimage feature extraction, so the model cannot be end-to-end pre-trained.
UniT [19] proposes amultimodal multi-task framework with a unified transformer; however, it can only support VQAand object detection tasks, but not captioning and grounding.
GPV [17] proposes a general-purposevision system, and FLAVA [57] presents a VL system similar to METER [13]; however, they didnot evaluate on grounding and detection tasks, and their performance on other VL tasks is still farfrom SoTA.
UNICORN [69] and OFA [64] reformulate grounding as a sequence generation task, byborrowing ideas from Pix2Seq [5].
However, these approaches have not been demonstrated to workon standard OD benchmarks, and also cannot be used as dual encoders for fast image retrieval.
Ourmodel is the first work that can support not only VQA, image captioning and O ( n + m ) retrieval, butalso visual grounding and object detection, with impressive performance across all tasks.
A detailedcomparison is provided in Table 1.
Method
In this section, we first describe the proposed model architecture in Section 3.1.
We then illustrateour two-stage pre-training paradigm in Section 3.2, followed by fine-tuning strategies for all the taskssupported by F IBER in Section 3.3.
3.1 Fusion in the Backbone
[Figure]
The architecture of F IBER is shown in Figure 2.
Differentfrom models that stack a modality fusion module on topof the vision or language backbones [6, 13], we insertmultimodal fusion inside the backbones, and include agating mechanism for the cross-modal layers (shown in
Figure 4).
Specifically, at each encoding layer, we have:
[Equation]
where α is a learnable parameter initialized to 0.
For simplicity, we insert the same number ofcross-attention layers into the vision and language backbones.
By inserting cross-attention layers with the gating mechanism, we enable cross-modal interactionswithout affecting the original computational flow of the backbones at the beginning of model training.
Also, we can easily switch off the interactions by setting α to 0, and the backbones can be used inthe dual-encoder setting.
In addition, compared to stacking a large number of transformer layers ontop of the backbones, our approach of inserting cross-attention layers is relatively light-weight andthus more memory-efficient.
To illustrate, both GLIP [34] and METER [13] use an additional 110Mmodality fusion parameters for a base-size model, while F IBER only adds about 26M parameters.
During training, the fusion module of F IBER only consumes half of the FLOPs needed by METER(12.35 vs. 24.04 GFLOPs for one instance).
We experimented with two other model variants forfusion in the backbone, the details of which are provided in Appendix.
3.2 Coarse-to-Fine Pre-training
We divide VL tasks into two categories based on whether or not we need to generate region-leveloutputs on the image side.
While these two kinds of tasks are characteristically different, they bothrequire fusion between the vision and language modalities, and we hypothesize that sharing as manyparameters as possible between the model used for these two sets of tasks will be beneficial.
Basedon this motivation, we propose a two-stage pre-training paradigm, where we first pre-train modelswith image-level objectives on images at low resolution, and then perform further pre-training withregion-level objectives where the input images are at a higher resolution.
In this way, the coarsegrained supervision from the first stage can provide good initialization for the second stage for all theshared parameters.
F IBER with the same architecture (Swin Transformer [42] and RoBERTa [41]) isused as the backbone for both stages of pre-training.
Coarse-grained Pre
-training.
For tasks like VQA and captioning, it has been demonstrated [32, 13,65] that masked language modeling (MLM), image-text matching (ITM), and image-text contrastive(ITC) objectives are helpful for ViT-based VLP models.
Following previous work, we use all thethree objectives during pre-training.
Specifically,
• For ITC , the inserted cross-attention modules are switched off, so F IBER functions as a dualencoder.
Given a batch of N image-caption pairs, we first compute their representations with
our vision and language encoders independently without modality fusion, and then maximize the
similarities between N positive image-text pairs while minimizing the similarities between the restN 2 − N negative pair, via a contrastive loss.
• For MLM and ITM , the inserted cross-attention modules are switched on, so F IBER now functionsas a fusion encoder.
For MLM, we randomly mask 15% of the input tokens and the model is trainedto reconstruct the original tokens.
For image-text matching, the model is given an image-textpair and predicts whether they are matched.
Following previous work [32, 65], we sample hard
negatives based on the similarities computed from the above ITC loss.
Fine-grained Pre-training.
Most existing VL architectures [6, 61, 25, 31, 64, 7] use vanillatransformers both for encoding the vision as well as language inputs.
However, in contrast to tokensin text, the entities of interest in images do not all occur at the same scale.
Being able to accuratelymodel the image at different scales is especially important for tasks such as object detection and phrasegrounding.
To handle this, it is typical in object detection literature to use input images at higherresolutions (800 × 1333), which becomes problematic when using vanilla transformers that scalequadratically in the length of the input sequence.
As mentioned earlier, we use a Swin Transformer[42] as our image encoder, which provides hierarchical representations of the image while havinglinear complexity in the size of the image.
We combine these multi-scale representations using an
FPN [38] for object detection training.
For fine-grained pre-training, we switch on the cross-attentionmodules, using F IBER as a fusion encoder.
This ensures that the image representations that are passedto the FPN are already text-aware, and is a crucial difference compared to GLIP [34], where theimage-text fusion takes place in the object detection head.
Once the text-aware image features areextracted by the Swin backbone and image-aware text features are extracted using RoBERTa [41], theimage features after the FPN are fed to a DynamicHead [9] which predicts a set of regions.
Just asin [34], we compute the dot product between the image region features R TA and the contextualizedtoken representations T IA to compute the grounding score:
[Equation]
where R TA represents regions that are text aware, produced using the OD-Head that takes as input
I TA , which are image representations that are already text-aware and T IA are the text features that have already attended to the image features.
The typical object detection model has a classificationhead that predicts the label of the object, and a localization head that predicts the bounding box.
We follow GLIP [34] by substituting the classification head with the grounding score S GROUNDING .
The localization loss is composed of two parts: a centerness loss and GIoU loss, which are used tosupervise the box prediction.
Taken together, F IBER learns the correspondence between regions in theimage and phrases in the text, making it possible to tackle tasks such as phrase grounding and objectdetection using the same framework.
We use ATSS framework [79] in our paper, but our method canbe combined easily with other object detectors such as Faster-RCNN [51] and RetinaNet [39] as well.
3.3 Adaptation to Downstream Tasks
We now describe how we adapt F IBER to different downstream tasks as depicted in Figure 3.
• For VL classification tasks such as VQA , we use F IBER as a fusion encoder.
Specifically, the top
M layers of the vision and language backbones interact with each other and produce multimodal
representations.
The final layer representations of the two modalities are concatenated together togenerate the final outputs for tasks such as VQA and visual reasoning.
• For retrieval tasks , we switch off the inserted cross-attention modules to use F IBER as a dualencoder for fast image-text retrieval.
• For captioning , we adapt F IBER by only keeping the image-to-text cross-attentions and usingcausal masks in the decoding side.
The representations of the final image encoding layer are fedinto the cross-attention modules.
In this way, the model is turned into a seq2seq model [60, 8] andperforms captioning in an auto-regressive way.
• For phrase grounding, object detection and referring expression comprehension , we use
F IBER as a fusion encoder, and the OD-Head introduced during fine-grained pre-training receivesimage features that are already language aware due to the multimodal representations extracted byF IBER .
The pre-trained model is directly used without any modifications for
these tasks.
Experiments
[Table]
Pre-training Datasets.
Following previouswork [6, 27, 32, 13, 63, 65], we perform coarsegrained pre-training on COCO [40], Conceptual Captions [55], SBU Captions [46], and Visual Genome [28].
The four datasets consist ofabout 4M images in total.
For fine-grained pretraining, we use two data sources: data curatedby MDETR [25] after removing the COCO images, and the Objects365 [54] detection dataset,together consisting of about 0.8M images.
Weensure that we exclude any data that exists inthe validation or test splits of downstream tasks.
Architecture.
We adopt Swin-Base [42] and
RoBERTa-Base [41] as our vision and text backbones, which are initialized with weights fromuni-modal pre-
training.
We insert cross-attention blocks into the top 6 blocks of the vision and textencoders.
The input resolution is 384 × 384 for coarse-grained pre-training and 800 × 1 , 333 forfine-grained pre-training.
Using a hierarchical vision transformer enables us to efficiently tackle thesehigh resolution tasks, which would be expensive in models such as BLIP [31] that rely on the vanillatransformer architecture.
In METER [13], which does explore using a Swin transformer as the imageencoder, the multi-modal fusion occurs in layers specifically designed to align the modalities, onlyafter the image and text features are extracted from the uni-modal backbones.
This is in contrast toour approach where the hierarchical image features that are used in the FPN for fine-grained trainingare already language aware, due to the multi-modal fusion being in the backbone.
This also letsus avoid adding additional “language-aware deep fusion layers” [34] as part of the OD head as in
GLIP, resulting in 1.5x faster training while maintaining performance as shown in Table 2.
While inprinciple it would be possible to use the image features extracted by METER’s backbone for objectdetection, it would be necessary as in GLIP to add additional layers to make the visual features“language-aware” for good detection performance, especially on datasets with limited training dataand with rare and infrequent objects.
[Table]
Implementation Details.
We perform coarse-grained pre-training for 100k steps with 4,096 batchsize on 64 A100 GPUs.
We use AdamW [43] with the peak learning rates of 1e-4 for the backbonesand 5e-4 for the cross-modal parameters.
We use linear warmup over the first 1k steps and lineardecay.
For fine-grained pre-training, we train for 800k steps on 64 V100 GPUs, with a batch size of64.
We use a learning rate of 1e-5 for the language backbone, and 1e-4 for the rest of the model witha weight decay of 0.01.
We use a linear warmup over the first 2k steps and then a constant learningrate, with two learning rate drops by a factor of 10 at 67% and 89% of the total number of steps.
4.1 Results on Downstream Tasks
Vision-Language Classification.
We first experiment on two representative VL classification tasks,including VQAv2 [3] and NLVR 2 [59].
As reported in Table 3, we achieve the best performancecompared to other models in the same setting.
It is worth noting that F IBER pre-trained with 4Mimages can achieve better performance than BLIP trained with 129M images and SimVLM trainedwith 1.8B images.
The results indicate that introducing fusion modules into the backbone is aneffective alternative to appending them on the top of uni-modal backbones.
Image-Text Retrieval.
While most previous VLP models perform retrieval by feeding every imagetext pair into the model, we remove the cross-modal modules in the backbones and measure thesimilarities by computing the dot products of image and text vectors computed separately.
This makesour approach much more efficient and scalable as each image and text needs to only be encodedonce.
As in Table 3, we can maintain competitive and sometimes even better retrieval performanceon both Flickr30k [48] and COCO [40] retrieval tasks.
This indicates the effectiveness of having theflexibility to switch on and off the cross-modal fusion modules.
Image Captioning.
We also evaluate our models on COCO [40] and NoCaps [1] captioning totest whether F IBER can be adapted to generation tasks.
As in Table 4, F IBER can achieve betterperformance than models trained on the same data with and without CIDEr optimization [52].
We findthat integrating GOLD [47] into F IBER can bring significant improvements, outperforming modelstrained with hundreds of millions of images.
Notably, we establish the absolute state-of-the-art
CIDEr scores on COCO for base-size models.
Considering that F IBER is not pre-trained to performcaptioning, the results demonstrate the strong generalization ability of F IBER .
Phrase Grounding.
Our fine-grained pre-training stage incorporates Flickr30k entities groundingdata, and we achieve 87.4 on the Recall@1 metric on the test set without any subsequent fine-tuning.
This not only surpasses the current SoTA [34] using a smaller sized model (Swin-B compared to their
[Table] [Table]
[Table] [Table]
[Table] [Table]
Swin-L), but also uses 25x less fine-grained data.
Our F IBER model is able to leverage the image-textcoarse-grained pre-training stage better, instead of relying on expensive pseudo-labelling of largeweb-scale corpus and subsequent high-resolution training on this generated fine-grained data as in[34].
We also compare our approach without using any coarse-grained VL training (image encoderinitialized to Swin-B weights from IN22k, and text encoder initialized to pre-trained RoBERTa), andeven in this setting, we are able to outperform a similarly sized GLIP model (GLIP-B), proving thatour fusion in the backbone is better at capturing fine-grained image-text understanding.
Referring Expression Comprehension (REC).
In contrast to many previous works [6, 15, 44] thattackle the REC task by re-ranking object proposals provided by an off-the-shelf detector, we follow[25] to directly predict the bounding box for the given referring expression.
Using our proposedtwo stage pre-training, F IBER achieves better performance than current SoTA [64] that uses a Largesized model.
Notably, on RefCOCOg
[75], which contains much longer referring expressions than in
RefCOCO/RefCOCO+ [26], we observe more than 2 points boost over OFA-L.
On the challengingtestB split of both RefCOCO and RefCOCO+, F IBER outperforms current SoTA, OFA-L.
Object Detection.
We report F IBER results on two standard object detection benchmarks, COCO[40] and LVIS [16], in zero-shot transfer 1 as well as fine-tuned settings in Table 7.
The LVIS dataset relevant data but it is not used for training for the task of interest.
For instance, our coarse-grained pre-training
[Table]
[Table] [Table]
- MDETR [25] - - /20.
- GLIP-T [34] 46.7/55.
17.7/ - 19.5/ - 31.0/ - 24.9/ - 44.4/63.9 GLIP-B [34] 48.1/57.
[Table]
Models pre- on and/or with
GLIP-L [34] 49.8/60.
F IBER -B 49.3/
[Table] [Table]
[Table] [Table] [Table]
[Table] [Table] [Table]
[Table] [Table] [Table]
[Table] [Table]
[Table]
consists of a long-tail of object classes, and is a popular test-bed for evaluating models on theirgeneralization capabilities and robustness to class imbalance.
On the APr metric, which is the Average
Precision on rare objects, F IBER outperforms GLIP-L which is a bigger model and also trained with25 × more fine-grained data.
[Figure]
We also report zero-shot and fine-tuned results on a suiteof 13 ODinW (object detection in the wild) datasets, spanning various domains and show consistent performanceimprovements over previous SoTA.
Additionally, in Figure 5, we report few-shot results aggregated across these13 datasets and show better data efficiency over GLIP-Btrained with the same fine-grained data.
Conclusion
We propose ( i ) F IBER , a novel architecture and ( ii ) acoarse-to-fine pre-training pipeline.
We perform extensive experiments and show consistent improvements overstrong baselines across a diverse set of tasks.
The resultsdemonstrate the effectiveness of F IBER coupled with our pre-training strategy, by setting new SoTAscores while at the same time reducing the requirement of expensive box-level annotations.
Futuredirections include scaling our models and extending our framework to other modalities.
The approach introduced in our work can potentially inherit undesirable societal biases that exist in our pre-training data.
Careful debiasing and filtering of data should be undertaken before real-life deployment of our work.
Additionally, pre-training can induce environmental costs, and minimizing these costs is an avenue that we plan to explore further
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See Section
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] SeeSection 5(d)
Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]2.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[No]
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes]
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[Yes]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]