Difference-Masking: Choosing What to Mask in Continued Pretraining (2023)

Abstract: The self-supervised objective of masked prediction has led to promising performance gains on a variety of downstream tasks. However, while most approaches randomly mask tokens, there is strong intuition that deciding what to mask can substantially improve learning outcomes. We investigate this in continued pretraining setting in which pretrained models continue to pretrain on domain-specific data before performing some downstream task. We introduce Difference-Masking, a masking strategy that automatically chooses what to mask during continued pretraining by considering what makes a task domain different from the pretraining domain. Empirically, we find that Difference-Masking outperforms baselines on continued pretraining settings across four diverse language-only and multimodal video tasks.

## introduction inspired by the distributional hypothesis in the language domain @xcite , masking is a selfsupervised learning (ssl) objective in which a model attempts to reconstruct hidden portions of data from the surrounding context. masking has enabled breakthrough performance on tasks from a variety of domains, such as language, vision, and speech @xcite @xcite , motivating interest in researching how masking strategies influence representation learning in ssl. masked prediction has recently been applied to adapt pretrained models to specific downstream tasks by continuing to pretrain models on indomain unlabelled data @xcite . masking in this continued pretraining setting been shown to be particularly effective when the target domain differs substantially from the pretraining domain @xcite . while prior work has studied how the amount masked influences model learning @xcite , most masking approaches randomly choose which parts of the data to mask. although it is understudied in ssl, deciding what to mask is a critical component in human education @xcite . educators designing "fill-in-the-blank" assessments for students must decide what content to mask in order to effectively assess student understanding of a domain @xcite . for example, in a real-world "fill-inthe-blank" chemistry test, a teacher might choose to mask domain-specific words ("density", "silicon") to assess student learning, instead of masking domain-irrelevant words ("example", "process"). in this paper, we propose difference-masking, a novel approach for automatically selecting what to mask during continued pretraining. our strategy first identifies anchors that describe what makes a target domain different from the pretraining domain and then determines what to mask during continued pretraining based on similarity to those anchors. in experiments spanning four diverse languageonly and multimodal video datasets (acl-arc, chemprot, tvqa, and social-iq), we find that difference-masking outperforms strong baselines, supporting our hypothesis that masking based on what is different about a task provides strong representation for continued pretraining. we provide intuitions to explain the strong performance of difference-masking, along with extensive 1 13222 analyses and ablations to better understand the performance of our method. our code is publicly available. ## related work masking relies on the distributional hypothesis, which posits that the meaning of a word can be inferred from its context @xcite . masking in nlp has functioned as an effective ssl strategy when training models such as bert @xcite and xl-net @xcite . although random masking has been more closely studied in nlp than non-random masking, there are three closely related works to ours from nlp. entitybert @xcite masks tokens based on whether they are part of "entities" recognized by a domain-specific pretrained namedentity-recognizer. salient span masking (ssm) @xcite ) is a similar method that uses a named-entity-recognition model to mask out a single entity for the downstream task of open-domain qa. however, these approaches require a domainspecific pretrained entity-tagger, and the masking strategy they determine is the same for any domain to which that tagger is applied. in contrast, difference-masking determines what to mask without pretrained entity-taggers, and its masking strategy can change depending on the unlabelled data in the task domain. selective masking @xcite uses data from the downstream task to decide which tokens to mask during continued pretraining by estimat-ing how much each token contributes to improved downstream task performance. it is important to note that selective masking uses supervised downstream task labels, whereas difference-masking is entirely self-supervised. prior work from the vision community has also contributed to an understanding of masking strategies, primarily by using the attention of the model during ssl training to determine what to mask. mst @xcite uses attention maps to determine "non-essential regions" to mask, while at-tnmask @xcite does the opposite by masking the most attended-to regions. unlike difference-masking, these approaches do not take into account domain-specific information when determining their masking strategy. this can be an impediment to performance when the model's attentions do not already contain information about what is important in a given input sequence. ## difference-masking this section describes the motivation and implementation of difference-masking: our selfsupervised method to determine a masking strategy for continued pretraining. the overall process is depicted visually in figure 2 . ## experimental settings our experiments evaluate whether difference-masking's masking strategy leads to performance improvements on challenging language-only and multimodal video understanding tasks. we follow the experimental setting from @xcite , in which unlabelled data from the downstream task domain is used for continued pretraining before eventually performing downstream task finetuning. this is a popular ssl setting because it represents a computationally-feasible way to test the effectiveness of self-supervised representation learning methods (e.g. without recreating a pretrained model), and it is realistic to modern approaches which rely heavily on pretrained models @xcite . experiments are performed to allow each model to learn as long as needed during continued pretraining, only stopping when validation error increases (early-stopping). each result is averaged across five random seeds. hyperparameter settings and data preprocessing details can be found in appendix a. ## conclusion in this paper we introduce difference-masking, a method for identifying what makes a target domain unique and using this information to guide a strategy that chooses what to mask during ssl continued pretraining. we find that our method outperforms strong baselines across diverse language and multimodal video understanding tasks. we provide a detailed discussion of what is masked in difference-masking and why our method performs well on various tasks. the crosstask applicability of difference-masking supports the effectiveness of our framework for ssl pretraining in language, vision, and other domains.