Curriculum Masking in Vision-Language Pretraining to Maximize Cross Modal Interaction (2024)

Abstract: Many leading methods in Vision and language (V+L) pretraining utilize masked language modeling (MLM) as a standard pretraining component, with the expectation that reconstruction of masked text tokens would necessitate reference to corresponding image context via cross/self attention and thus promote representation fusion. However, we observe that the minimization of MLM loss in earlier training stages can depend disproportionately on local text signals, leading to poor training efficiency and inconsistency with the goal of representation fusion. The extent of this lack of cross modal interaction depends strongly which token(s) are masked. To address this issue, we propose a curriculum masking scheme as a replacement for random masking. Tokens are selected to be masked at a frequency proportional to the expected level of cross modal interaction necessary to reconstruct them. This is achieved using a parallel mask selection agent that measures the cross modal flow of information and treats it as a reward to be maximized. By additionally masking contiguous spans that include key objects and their relations, we also achieve better relational understanding, which has been shown to be lacking in many SOTA models. Our experiments on a wide range of V+L tasks show that we trail closely behind state-of-the-art methods despite pretraining on 300x to 1000x less data and we also achieve either top or runner-up performance on tasks from the ARO benchmark which tests compositional relationships. Finally, we demonstrate the potential of our method to scale to larger pretraining data.

## introduction in the short time since large scale pretraining was first introduced to the multimodal setting @xcite @xcite , performance on leading benchmarks has risen by up to ten percentage points @xcite @xcite , driven in no small part by the trend towards larger, often custom curated image-text pair datasets @xcite . however, the topic of pretraining efficiency has not received the attention it deserves. masked language modeling is widely used as a pretraining component @xcite @xcite , with the expectation that reconstruction of masked text tokens would necessitate reference to corresponding image context via cross/self attention and thus promote representation fusion. the issue is that resulting cross modal interaction is highly dependent on the token that is masked. from an intuitive perspective, some tokens-such as prepositions and stop words (1)-can be reconstructed entirely disregarding image context, while others (2) can be narrowed down to a few possibilities from surrounding text alone. as can be seen in figure 1 , when we replace paired images with random ones and calculate cosine similarities between the fused representations of masked tokens for original text-image pairs and their random-image variants, this swapped context similarity (scs) score remains relatively static for many epochs at the start even though mlm loss rapidly decreases. it is not until a third of the way through training that image context noticeably affects learned representations. the large variance in scs score at the start also reflects on how mask selection affects cross modal interaction. clearly, alternatives to random masking warrant exploration. in the interest of pretraining efficiency, (1) could be masked less often compared to tokens requiring targeted attention to image context to reconstruct. (2) would ideally be masked less early on, until the model learns to venture beyond local textual context. while several adaptive masking strategies have been proposed (, levine2020pmi, yang2022learning) they apply to the unimodal case and cannot be directly adapted to maximize cross modal interaction. another pressing issue is that many advanced models fail to demonstrate relational understanding; many were recently shown to achieve at-or-below-chance performance at discerning between captions like "the horse is eating the grass" and "the grass is eating the horse" @xcite , despite their rich pretraining data. their behavior was likened to "a bag of words model", suggesting the lack of nuanced cross modal information flow. a particular property of transformer architectures can be exploited to measure cross modal interaction and help address these issues-the attention between text and visual tokens, based on which we formulate an interaction score. since we hope to maximize this score over mask selection rather than model parameters, we cannot set it up as an optimization problem baked into the loss function etc. we thus treat the problem as a reinforcement learning one, with mask configuration as the action space and the interaction score as the reward, and use a parallel model with shared parameters to select masks while the main model performs multimodal mlm. this curriculum masking strategy also includes masking contiguous spans and learned factorization order, making it a generalized form of language modeling, of which masked and autoregressive language modeling are special cases. we find that this strategy results in not only more data-efficient pretraining, but also a natural solution to the issue of relational understanding, as reconstruction of a masked spans that include key objects and their relations forces relational learning. our contributions are as follows: there is no delay as the parallel network performs masking/order selection for batch t + 1 while the main model learns from batch t. our masking strategy is also compatible with most multimodal transformer architectures and can serve as a complement to concurrent works in multimoddal pretraining. ## conclusion in this paper, we present a curriculum masking strategy for v+l pretraining. masks and factorization orders are selected by a parallel agent that aims to maximize cross modal interaction. better pretraining efficiency and relational understanding are achieved, as demonstrated on various downstream tasks and experiments, at a reasonable speed and memory cost. the proposed method is also able to complement concurrent methods in v+l pretraining.