NB - MLM : Efficient Domain Adaptation of Masked Language Models for Sentiment Analysis (2021)

Abstract: While Masked Language Models (MLM) are pre-trained on massive datasets, the additional training with the MLM objective on domain or task-specific data before fine-tuning for the final task is known to improve the final performance. This is usually referred to as the domain or task adaptation step. However, unlike the initial pre-training, this step is performed for each domain or task individually and is still rather slow, requiring several GPU days compared to several GPU hours required for the final task fine-tuning. We argue that the standard MLM objective leads to inefficiency when it is used for the adaptation step because it mostly learns to predict the most frequent words, which are not necessarily related to a final task. We propose a technique for more efficient adaptation that focuses on predicting words with large weights of the Naive Bayes classifier trained for the task at hand, which are likely more relevant than the most frequent words. The proposed method provides faster adaptation and better final performance for sentiment analysis compared to the standard approach.

## introduction pre-training of neural networks with a language model (lm) or masked language model (mlm) objective on large amounts of non-domain-specific texts has given a significant boost of performance in almost all natural language processing tasks. while 16gb of texts were shown to bert @xcite and ten times more to roberta @xcite during pre-training, the further training of these models with the mlm objective on domainspecific texts before fine-tuning to the target task was shown to further improve the final results @xcite . this technique is called the domain or task adaptation, depending on the degree of similarity of the data for adaptation to the target dataset. while initial pretraining is extremely expensive, it does not depend on the final task and can be performed only once. however, domain or task adaptation is done for each domain or task individually and is still quite resource-demanding, requiring hundreds of thousands of training steps or several gpu days, unlike final fine-tuning, which can often be done in a few gpu hours @xcite . in this work, we propose a method for more efficient mlm adaptation. we have noticed that the standard mlm spends most of the training time on learning to restore the most frequent words like determiners or auxiliary verbs hidden (masked) from its input. while such training examples may be useful for learning english grammar, their domination during the adaptation phase seems to be wasteful for many final tasks. since the final task and the dataset are already known in this phase, we propose to undersample such examples in favor of examples with targets related to the final task. this relatedness is estimated using a naive bayes classifier. hence, we call our modified objective naive bayes masked language model (nb-mlm). we hypothesize that hiding from the model and asking it to restore mostly features that are important for the final task will likely result in faster adaptation. additionally, the absence of simple features and the requirement to restore them may teach the model to exploit more sophisticated and implicit features relevant to the final task. we evaluate the proposed method on two datasets for sentiment analysis. it is one of the most popular tasks in natural language processing @xcite and an excellent playground for the comparison of adaptation methods due to the large amount of labeled and unlabeled user reviews of different products available. in particular, we consider the task of classifying the binary sentiment polarity of a given review. our experiments 0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 percent of the target tokens the , . a and of to is br in it i that this 's was movie /> >< / tokens uniform 0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 percent of the target tokens bad ? great love best acting plot well nothing no thing script minutes worst always performance 2 horror 't money nb-mlm show that the nb-mlm objective can significantly reduce adaptation time while achieving the same final performance or help to improve performance given the same amount of time for adaptation. foot_0 2 related work pre-training transformer networks with the mlm objective is proposed in @xcite for the bert model and is shown to outperform the more traditional lm objective, though the similar task of predicting a word from its left and right context was used with different architectures earlier @xcite . roberta enhances bert by pre-training longer on ten times larger corpora, getting rid of the next sentence prediction (nsp) task during pre-training, and selecting different target words to be masked and predicted in each epoch (dynamic masking). various approaches to further pre-training of bert on domain or task-specific data are compared in @xcite , while @xcite carry out a similar investigation with roberta. they try various options of data sources for adaptation: texts only from the target dataset (called task adaptation or within-task pre-training), larger datasets from the same domain (called domain adaptation or in-domain pretraining), and datasets from different domains (called cross-domain pre-training). they find the task adaptation, which is a computationally cheapest option, to be a surprisingly good solution. in their experiments, it often outperforms the domain adaptation and is only marginally worse than com-bining both methods. however, due to the large amount of data used in domain adaptation, @xcite train the mlm only for one or very few epochs. we find that our method leveraging large data more efficiently makes the domain adaptation comparable to the task adaptation, and their combination is significantly better than each of them. our idea of employing naive bayes weights is inspired by the nb-svm model @xcite , which scales bag-ofngrams vectors with naive bayes classifier weights and then trains linear svm or logistic regression classifiers on them. it proved to be a very strong baseline, often outperforming both linear and more sophisticated models from that time. ## mlm objectives for adaptation uniform mlm. for each input example, the standard mlm objective, as proposed by @xcite , samples 15% of the input positions (subwords) for calculating the loss. the positions are sampled from the uniform distribution without replacement: p (pos) ∝ 1. then 80% of the tokens on sampled positions are masked (replaced with a [mask] token), 10% are replaced with some random tokens from the uniform distribution over the vocabulary, and 10% are left intact. ## experiments and results during the preliminary experiments described in appendix a, we found that our method helps for both bert and roberta models. however, the latter model achieved significantly better performance. therefore, we describe the results for roberta in the rest of the paper. for domain adaptation (denoted as dapt), we employed the amazon reviews dataset @xcite with duplicates removed. we removed reviews shorter than 500 characters and split the rest into the training and validation sets of 21m and 10k reviews correspondingly. the valida- for domain and task adaptation, we used the batch size of 1024, while classifiers were fine-tuned with the batch size of 32. based on our preliminary experiments, we set the learning rate of 2e-4 for the domain adaptation, 1e-4 for the task adaptation, and 1e-5 for final fine-tuning. following gururangan et al. (2020), we performed domain adaptation for one epoch on the amazon dataset (20k steps, 38h on one v100 gpu) and task adapta-2 using the whole target dataset for task adaptation has shown the best results for both uniform mlm and nb-mlm, see appendix c. this setup, when test examples (without labels) are exploited during training, is known as transductive learning. figure 2 shows how the final classification accuracy improves during the task and domain adaptation. our nb-mlm model significantly helps for domain adaptation on imdb. for task adaptation, the difference is much smaller and fits into two standard deviations. still, on average, the nb-mlm seems to provide a consistent improvement throughout the adaptation. for yelp, the improvements from nb-mlm are also small but consistent. for imdb, the domain adaptation with nb-mlm obtains results similar to the uniform mlm in 5x fewer training steps and data (only 20% of the data is seen during the first 4k steps). when trained for one epoch, it improves the results by more than 0.3%, which is also statistically significant. for task adaptation, the nb-mlm gives a much smaller improvement. similarly to the results of @xcite , in our experiments, the task adaptation with the uniform mlm outperforms the domain adaptation that employs much more data by almost 0.5%. we suppose that this is due to the small proportion of relevant examples sampled by the uniform mlm, which require many repetitions to learn from. probably, training domain adaptation for hundreds of epochs, similarly to task adaptation, can fix this problem, but this is not feasible for large datasets and moderate computation resources. more efficient domain adaptation with nb-mlm, which focuses on targets that are likely relevant for the final task, reduces this difference to 0.2%. finally, using the domain adaptation followed by the task adaptation results in the best final performance. in this scenario, nb-mlm gives 0.2% improvement for short adaptation and 0.1% for long adaptation. for yelp, the metrics are higher, and the differences are smaller but still consistent. ## conclusion we proposed a technique for the more efficient domain and task adaptation of mlms. it is especially helpful for leveraging large data efficiently during the domain adaptation, resulting in significantly shorter adaptation time or better performance.