Train No Evil: Selective Masking for Task-Guided Pre-Training (2020)

Abstract: Recently, pre-trained language models mostly follow the pre-train-then-fine-tuning paradigm and have achieved great performance on various downstream tasks. However, since the pre-training stage is typically task-agnostic and the fine-tuning stage usually suffers from insufficient supervised data, the models cannot always well capture the domain-specific and task-specific patterns. In this paper, we propose a three-stage framework by adding a task-guided pre-training stage with selective masking between general pre-training and fine-tuning. In this stage, the model is trained by masked language modeling on in-domain unsupervised data to learn domain-specific patterns and we propose a novel selective masking strategy to learn task-specific patterns. Specifically, we design a method to measure the importance of each token in sequences and selectively mask the important tokens. Experimental results on two sentiment analysis tasks show that our method can achieve comparable or even better performance with less than 50% of computation cost, which indicates our method is both effective and efficient. The source code of this paper can be obtained from https://github.com/thunlp/SelectiveMasking .

## introduction pre-trained language models (plms) have achieved superior performances on various nlp tasks @xcite @xcite @xcite and have attracted wide research interests. inspired by the success of gpt @xcite and @xcite , most plms follow the pre-train-then-fine-tuning paradigm, which adopts unsupervised pre-training on large general-domain † corresponding author: z.liu (liuzy@tsinghua.edu.cn) corpora to learn general language patterns and supervised fine-tuning to adapt to downstream tasks. recently, @xcite shows that learning domain-specific and task-specific patterns during pre-training can be helpful to the models for certain domains and tasks. however, conventional pre-training is aimless with respect to specific downstream tasks, and fine-tuning usually suffers from insufficient supervised data, preventing plms from effectively capturing these patterns. to learn domain-specific language patterns, some previous works @xcite pre-train a bert-like model from scratch using large-scale in-domain data. however, they are computation-intensive and require largescale in-domain data, which is hard to obtain in many domains. to learn task-specific language patterns, some previous works @xcite add intermediate supervised pre-training after general pre-training, whose pre-training task is similar to the downstream task but has a larger dataset. however, @xcite shows that this kind of intermediate pre-training often negatively impacts the transferability to downstream tasks. to better capture domain-specific and taskspecific patterns, we propose a three-stage framework by adding a task-guided pre-training stage with selective masking between the general pretraining and fine-tuning. the overall framework is shown in figure 1 . in the stage of task-guided pre-training, the model is trained by masked language modeling (masked lm) @xcite on mid-scale in-domain unsupervised data, which is constructed by collecting other corpora in the same domain. in this way, plms can utilize more data to better learn domain-specific language patterns @xcite @xcite @xcite . however, the conventional masked lm randomly masks tokens, which is inefficient to learn task-specific language patterns. hence, we propose a selective masking strategy for task-guided pretraining, whose main idea is selectively masking the important tokens for downstream tasks. intuitively, some tokens are more important than others for a specific task and the important tokens vary among different tasks @xcite @xcite . for instance, in sentiment analysis, sentiment tokens such as "like" and "hate" are critical for sentiments classification @xcite . and, in relation extraction, predicates and verbs are typically more significant. if plms can selectively mask and predict the important tokens instead of a mass of random tokens, they can effectively learn task-specific language patterns and the computation cost of pretraining can be significantly reduced. for the selective masking strategy, we propose a simple method to find important tokens for downstream tasks. specifically, we define a task-specific score for each token and if the score is lower than a certain threshold, we regard the token as important. however, this method relies on the supervised downstream datasets whose sizes are limited for pre-training. to better utilize mid-scale in-domain unsupervised data as shown in figure 1 , we train a neural network on downstream datasets where the important tokens are annotated using the method mentioned above. this neural network can learn the implicit token-selecting rules, which enables us to select tokens without supervision. we conduct experiments on two sentiment analysis tasks: mr @xcite and semeval14 task 4 @xcite . experimental results show that our method is both efficient and effective. our method can achieve comparable and even better performances than the conventional pre-trainthen-fine-tune method with less than 50% of the overall computation cost. ## methodology in this section, we describe task-guided pretraining and selective masking strategy in detail. for convenience, we denote general unsupervised data, in-domain unsupervised data, downstream supervised data as d general , d domain and d task . they generally contain about 1000m words, 10m words, and 10k words respectively. ## conclusion in this paper, we design task-guided pre-training with selective masking and present a three-stage training framework for plms. with task-guided pre-training, models can effectively and efficiently learn domain-specific and task-specific patterns, which benefits downstream tasks. experimental results show that our methods can achieve better performances with less computation cost. note that although we only conduct experiments on two sentiment classification tasks using bert as the base model, our method can easily generalize to other models using masked language modeling or its variants and other text classification tasks. besides, there are still two important directions for future work: (1) how to apply task-guided pre-training to general domain data when the indomain data is limited. (2) how to design more effective strategies to capture domain-specific and task-specific patterns for selective masking.