Multimodal Pretraining Unmasked: A Meta-Analysis and a Unified Framework of Vision-and-Language BERT s (2021)

Abstract: Large-scale pretraining and task-specific fine- tuning is now the standard methodology for many tasks in computer vision and natural language processing. Recently, a multitude of methods have been proposed for pretraining vision and language BERTs to tackle challenges at the intersection of these two key areas of AI. These models can be categorized into either single-stream or dual-stream encoders. We study the differences between these two categories, and show how they can be unified under a single theoretical framework. We then conduct controlled experiments to discern the empirical differences between five vision and language BERTs. Our experiments show that training data and hyperparameters are responsible for most of the differences between the reported results, but they also reveal that the embedding layer plays a crucial role in these massive models.

## introduction learning generic multimodal representations from images paired with sentences is a fundamental step towards a single interface for vision and language (v&l) tasks. in pursuit of this goal, many pretrained v&l models have been proposed in the last year, inspired by the success of pretraining in both computer vision @xcite and natural language processing @xcite . all of these v&l models extend bert @xcite to learn representations grounded in both modalities. they can either be classified as (i) single-stream, where images and text are jointly processed by a single encoder (e.g., @xcite , or (ii) dual-stream, where the inputs are encoded separately before being jointly modelled (e.g., @xcite . the differences in downstream performance between single-and dual-stream models are cur-rently unclear, with some papers claiming the superiority of one family over the other @xcite , while others arguing that it is hard to draw any conclusion @xcite . the first goal of this paper is to understand the mathematical differences between single-and dual-stream models. our analysis leads to a unified framework in which currently proposed architectures, both single-and dual-stream, are particular instances. we then implement several of the proposed encoders within this framework to empirically measure their differences in a controlled environment. we believe this comparative analysis is crucial to better understand and guide future research of massive models in this vibrant area of ai, ensuring progress is not blurred by confounds. in fact, there are many differences in the protocols used to train v&l berts. in order to better understand these models, we conduct a series of controlled studies to investigate whether differences in downstream performance is explained by: (i) the amount of pretraining data and the pretraining objectives (e.g., figure 2 ); (ii) the hyperparameters used to control the learning process; (iii) the variance caused by random initialization when pretraining (e.g., figure 1 ); (iv) the variance due to fine-tuning multiple times on a downstream task; (v) being single-or dual-stream architectures; or (vi) the choice of the embedding layer. in summary, our contributions in this paper are: figure 1 : how does the amount of pretraining data affect downstream performance of v&l berts? we find that these models perform more similarly when trained in the same conditions. this plot shows the results from the papers (♦), and when each model is pretrained 10 times on the conceptual captions dataset and fine-tuned once on the nlvr2 verification task (•). the area of a marker is proportional to the amount of pretraining data. the result from the visualbert paper is highlighted in a dashed box. ## vision-and-language berts given a sequence of tokens {w 1 , . . . , w t } and a set of visual features {v 1 , . . . , v k }, a shared goal of v&l bert models is to produce cross-modal representations that are useful for downstream tasks grounded in both modalities. in this section, we first review how these models embed their inputs to the feature space. next, we discuss the main differences in the encoders and, finally, highlight a variety of confounds that might affect the performance achieved by these models. ## a unified framework in this section, we unify the recently proposed single-stream and dual-stream architectures under the same mathematical framework. we start by reviewing the transformer layer, which forms the core of these architectures, then we explain how this layer has been adapted to encode multimodal data in v&l berts, and introduce a gated bimodal transformer layer that implements all of the architecture variants as special cases. ## experimental setup in this section, we present the experimental setup for our controlled studies on v&l encoders. volta in order to facilitate research and development of v&l pretraining, we release volta (visiolinguistic transformer architectures), an implementation of our unified framework in py-torch @xcite . our code is built on top of the vilbert-mt repository, 7 based on pytorch-transformers, due to its support to a wide range of v&l tasks. we stress that it is important, for this study, to have a unified implementation that allows us to remove possible confounds due to implementation details and effectively measure differences given by the proposed architectures. implementation details v&l berts typically extract image features using a faster r-cnn @xcite trained on the visual genome dataset (vg; @xcite , either with a resnet-101 @xcite or a resnext-152 backbone @xcite . the number of features varies from 10 to 100. our models are trained with 36 regions of interest extracted by a faster r-cnn with a resnet-101 backbone @xcite . each model is initialized with the parameters of bert, following the approaches described in the original papers. 8 randomly initialized weights are initialized following the standard approach in pytorch-transformers (on which these models built on): fully-connected and embedding layers are initialized from a normal distribution with mean 0.0 and standard deviation 0.02, bias vectors are initially set to 0.0, and the layer normalization weight vector to 1.0. we train all models on 4 nvidia p100 gpus and rely on gradient accumulation to obtain larger batches when needed. the parameter sets giving the best validation performance based on the pretraining objective are used for downstream tasks. pretraining as discussed in §2.4, v&l berts have been pretrained on datasets of varying size and type. 9 in this paper, we pretrain all of our models on the conceptual captions dataset (cc; @xcite ), which consists of 3.3m images with weakly associated captions automatically collected from billions of web pages. this stands in contrast to other datasets, for example, coco @xcite or vqa @xcite , where the images are strongly associated with crowdsourced captions or question-answer pairs. the cc dataset is a good candidate for learning generic multimodal representations because of its size, that it was scraped from the web, and that it has a broad coverage of subject matter. 10 note that due to broken links, and a subsequent pruning phase, where images also found in the test sets of 8 only @xcite reported slightly better performance when pretraining from scratch but they relied on large corpora of in-domain, human-annotated data. ## results we perform carefully controlled experiments to investigate the possible reasons for the reported difference in performance between v&l berts. ## reproducibility and the environment from the perspective of reproducible research, there are several advantages to using the volta framework for v&l encoders. first, volta reduces confounds due to differences in implementations, while also enabling fair comparisons with related work. second, visual and textual data only need to be preprocessed once instead of creating model-specific formats for every v&l bert. from a financial perspective, the costs involved in pretraining hampers contributions from many academic institutions and deters the evaluation of multiple trained models, which we showed to be extremely important for v&l berts. we estimate that pretraining a single model 10× in our controlled setup for 4 downstream tasks requires a 4-gpu machine on aws for two months, at a cost of ∼$6,000, corresponding to 200 gpu-compute days. fortunately, we had access to an internal server, but our experiments still required 1,500 gpu days for training and evaluation. while we were able to reduce the financial costs, there are severe environmental and carbon footprint costs in v&l pretraining @xcite . 18 we hope that volta will serve as a basis for research in v&l pretraining, enabling easy and fair comparisons across architectures, and ensuring that progress is not obfuscated by confounds. ## conclusion we introduced and implemented a unified mathematical framework, under which recently pro18 we distribute many of our pretrained v&l berts in volta to amortise the environmental costs. our fast-paced field rewards the contribution of new methods and state-of-the-art results @xcite , which often contrasts with controlled comparisons and training multiple models for variance estimation. in this paper, we showed that several methods for vision-andlanguage representation learning do not significantly differ when compared in a controlled setting. this finding echoes similar studies of variants of lstms @xcite and transformers @xcite that are not significantly better than the original models. looking to the future, we recommend that new v&l berts are pretrained on similar datasets, and that researchers report fine-tuning variance, in addition to their best performing model. we hope that our findings will encourage more controlled evaluations of newly proposed architectures for visionand-language and beyond. european union's horizon 2020 research and innovation programme under the marie skłodowska-curie grant agreement no. 801199 and by ''research and development of deep learning technology for advanced multilingual speech translation,'' the commissioned research of national institute of information and communications technology (nict), japan.