I nfor M ask: Unsupervised Informative Masking for Language Model Pretraining (2022)

Abstract: Masked language modeling is widely used for pretraining large language models for natural language understanding (NLU). However, random masking is suboptimal, allocating an equal masking rate for all tokens. In this paper, we propose InforMask, a new unsupervised masking strategy for training masked language models. InforMask exploits Pointwise Mutual Information (PMI) to select the most informative tokens to mask. We further propose two optimizations for InforMask to improve its efficiency. With a one-off preprocessing step, InforMask outperforms random masking and previously proposed masking strategies on the factual recall benchmark LAMA and the question answering benchmark SQuAD v1 and v2.

## introduction masked language modeling (mlm) is widely used for training language models @xcite @xcite . mlm randomly selects a portion of tokens from a text sample and replaces them with a special mask token (e.g., @xcite ). however, random masking has a few drawbacks -it sometimes produces masks that are too easy to guess, providing a small loss that is inefficient for training; some randomly masked tokens can be guessed with only local cues @xcite ; all tokens have an identical probability to be masked, while (e.g.) named entities are more important and need special attention @xcite . in this paper, we propose a new strategy for choosing tokens to mask in text samples. we aim to select words with the most information that can benefit the language model, especially for knowledge-intense tasks. to tackle this challenge, we propose informask, an unsupervised informative masking strategy for language model pretraining. first, we introduce informative relevance, a metric based on pointwise mutual information @xcite to measure the quality of a masking choice. optimizing this measure ensures the informativeness of the masked token while maintaining a moderate difficulty for the model to predict the masked tokens. this metric is based on the statistical analysis of the corpus, which does not require any supervision or external resource. however, maximizing the total informative relevance of a text sample with multiple masks can be computationally challenging. thus, we propose a sample-and-score algorithm to reduce the time complexity of masking and diversify the patterns in the output. an example is shown in figure 1 . for training a language model with more epochs, we can further accelerate the masking process by only running the algorithm once as a preprocessing step and assigning a token-specific masking rate for each token according to their masking frequency in the corpus, to approximate the masking decisions of the sample-and-score algorithm. after this one-off preprocessing step, masking can be as fast as the original random masking without any further overhead, which can be desirable for largescale distributed language model training of many epochs. to verify the effectiveness of our proposed method, we conduct extensive experiments on two knowledge-intense tasks -factual recall and question answering. on the factual recall benchmark lama @xcite , informask outperforms other masking strategies by a large margin. also, our base-size model, informbert, trained with the same corpus and epochs as bert @xcite outperforms bert-base on question answering benchmark squad @xcite . notably, on the lama benchmark, informbert outperforms bert and thomas edison was an inventor and businessman. thomas edison was an inventor and businessman. thomas edison was an inventor and businessman. thomas edison was an inventor and businessman. thomas edison was an inventor and businessman. to summarize, our contributions are as follows: informbert also achieves competitive results on the question answering datasets squad v1 and v2. words that have high informative value may be masked less frequently compared to (e.g.) stop words, given their frequencies in the corpus. ## methodology informask aims to make masking decisions more 'informative'. since not all words are equally rich in information @xcite , we aim to automatically identify more important tokens (e.g., named entities) and increase their probability to be masked while preserving the factual hints to recover them. on the other hand, we would like to reduce the frequency of masking stop words. stop words are naturally common in the corpus and they can be important for learning the syntax and structure of a sentence. however, masked stop words can be too easy for a language model to predict, especially in later stages of lm pretraining. thus, properly reducing the masking frequency of stop words can improve both the efficiency and performance of the model. ## conclusion in this work, we propose informask, an unsupervised masking policy that masks tokens based on their informativeness. informask achieves superior performance in knowledge-intense tasks including factual recall and question answering. we explore the impact of different masking strategies on learning factual and commonsense knowledge from pretraining and analyze why previously proposed masking techniques are suboptimal. for future work, we would like to scale up the pretraining and explore more factors for knowledge acquisition during unsupervised text pretraining.