Data Efficient Masked Language Modeling for Vision and Language (2021)

Abstract: Masked language modeling (MLM) is one of the key sub-tasks in vision-language pretraining. In the cross-modal setting, tokens in the sentence are masked at random, and the model predicts the masked tokens given the image and the text. In this paper, we observe several key disadvantages of MLM in this setting. First, as captions tend to be short, in a third of the sentences no token is sampled. Second, the majority of masked tokens are stop-words and punctuation, leading to under-utilization of the image. We investigate a range of alternative masking strategies specific to the cross-modal setting that address these shortcomings, aiming for better fusion of text and image in the learned representation. When pre-training the LXMERT model, our alternative masking strategies consistently improve over the original masking strategy on three downstream tasks, especially in low resource settings. Further, our pre-training approach substantially outperforms the baseline model on a prompt-based probing task designed to elicit image objects. These results and our analysis indicate that our method allows for better utilization of the training data.

## introduction pre-trained vision-language (vlp) models such as vilbert @xcite , lxmert @xcite and uniter @xcite have recently improved the state-of-the-art across various vision and language benchmarks. one of the primary pre-training objectives of vlp is masked language modeling (mlm). motivated by the single-modal mlm task, most models perform as introduced in bert @xcite ) for text-only data, randomly masking tokens with a probability of 15% @xcite . figure 1 : illustration of our approach. the baseline mlm masks a random token with 15% probability, where ≈50% of the masked tokens are stop-words or punctuation. our method masks words that require the image in order to be predicted (e.g., physical objects). our pre-train masking strategy consistently improves over the baseline strategy in two evaluation setups. the main difference in the cross-modal setting foot_1 is that the model takes into account both the textual context and the image, and the latter can help it resolve ambiguities. for example, in figure 1 , given the masked sentence "a [mask] is eating the carrot", without the image, the model might predict rabbit, since it is correlated with carrot. but the image reveals that the answer is tiger. in this work, we find that the mlm pre-training method is sub-optimal for vlp, as it does not make efficient use of the training data. this manifests in two major shortcomings, common to many popular pre-train datasets @xcite @xcite . first, we observe that image captions, which form the textual part of these corpora, tend to be much shorter than the documents in bert's pre-train data. as a result, uniformly masking tokens at 15% probability results in many cases where no token is being masked (e.g., about one third in lxmert). second, we note that 45%-50% of the masked tokens are stop-words or punctuation. while this seems a common phenomena also in text-only datasets, we show that this causes the image to be under-used in mlm pre-training for vlp. evidently, for the popular lxmert model, we find that the mlm validation accuracy on stop-words and punctuation is almost perfect even when omitting the image. to address these limitations, we propose alternative strategies aiming to mask words that require the image (e.g., physical objects). we pre-train the lxmert model with these strategies and demonstrate their benefits in two evaluation setups. first, on three vlp downstream tasks (gqa, @xcite @xcite , our masking strategies consistently improve over the traditional mlm, especially in low resource settings. second, we experiment with prompt based object detection @xcite , a probing task designed to elicit image objects by presenting the pre-trained models with prompts such as "a photo of [mask]" and compare their top predictions with image objects. our results show that our strategy substantially improves over the baseline sampling approach, even when trained over only a third of its epochs and half of its training data. in our analysis, we introduce a new metric (∆ image loss) to estimate the necessity of the image for a masked word during mlm. we extract the ∆ image loss value for each token in lxmert validation pre-train data. we then present a hierarchy of semantic classes ranked by this metric, and find that the frequently masked tokens in our strategies indeed increase the image necessity. our main contributions are: (1) we show that the current mlm pre-training method is sub-optimal for vlp, and it does not make efficient use of pretrain data. (2) we propose alternative masking strategies, and show that models trained with these strategies outperform the baseline strategy in two evaluation setups, especially in low resource settings. (3) we introduce the ∆ image loss metric, which aims to explain the relation between a masked token and the image; we publicly release the computed values of this metric for the lxmert validation set; this data may be used in future work to devise improved masking strategies. ## limitations of mlm approaches for vision and language in this section, we present the limitations of the mlm approach to vision and language tasks. we start by reviewing the way mlm is currently applied in cross-modal models, and analyzing the pretrain datasets used by most models. we observe the following two major limitations in the current approach: (1) no token is masked in roughly a third of the sentences; (2) a substantial part of the masked tokens are stop-words or punctuation, which can be predicted based on textual context alone, and do not require the image. ## alternative masking strategies to overcome the limitations presented in the previous section, we introduce several alternative masking strategies for cross-modal mlm. the proposed strategies use several semantic classes, which are introduced in section 3.1, and then used in section 3.2. ## experiments to evaluate the value of our proposed strategies, we conduct experiments by pre-training models with different masking strategies and evaluate them on 7 of the three words with the highest concreteness value in the sentence, mask the most concrete word with 55% probability, the second most concrete with 30% probability, and the third most with 15% probability. two evaluation setups. we describe the experimental setups below. ## conclusions we have shown that the current mlm pre-training method is sub-optimal for visual language pretraining, as this process tends to focus on stop words and punctuation, and in many cases does not mask any word in the sentence. we proposed alternative masking strategies that better utilize the image during pre-training, for example, focusing on physical objects. we found improved results in two evaluation setups, especially in low resource settings. we introduced the ∆ image loss metric, which aims to explain the relation between a masked token and the image. our analysis includes a hierarchy that describes the necessity of the image for different semantic classes. we publicly release the extracted data with this metric on the lxmert pre-train validation data. future work can use this information to devise new masking strategies, and progress towards vlp models that better leverage the visual aspect of the cross-modal tasks.