Masking as an Efficient Alternative to Finetuning for Pretrained Language Models (2020)

Abstract: We present an efficient method of utilizing pretrained language models, where we learn selective binary masks for pretrained weights in lieu of modifying them through finetuning. Extensive evaluations of masking BERT, RoBERTa, and DistilBERT on eleven diverse NLP tasks show that our masking scheme yields performance comparable to finetuning, yet has a much smaller memory footprint when several tasks need to be inferred. Intrinsic evaluations show that representations computed by our binary masked language models encode information necessary for solving downstream tasks. Analyzing the loss landscape, we show that masking and finetuning produce models that reside in minima that can be connected by a line segment with nearly constant test accuracy. This confirms that masking can be utilized as an efficient alternative to finetuning.

## introduction finetuning a large pretrained language model like bert @xcite , roberta @xcite , and xlnet @xcite often yields competitive or even state-of-the-art results on nlp benchmarks @xcite . given an nlp task, standard finetuning stacks a linear layer on top of the pretrained language model and then updates all parameters using mini-batch sgd. various aspects like brittleness @xcite and adaptiveness @xcite of this two-stage transfer learning nlp paradigm @xcite have been studied. despite the simplicity and impressive performance of finetuning, the prohibitively large number of parameters to be finetuned, e.g., 340 million in bert-large, is a major obstacle to wider deployment of these models. the large memory footprint of finetuned models becomes more prominent * equal contribution. when multiple tasks need to be solved -several copies of the millions of finetuned parameters have to be saved for inference. recent work @xcite points out the potential of searching neural architectures within a fixed model, as an alternative to optimizing the model weights for downstream tasks. inspired by these results, we present masking, a simple yet efficient scheme for utilizing pretrained language models. instead of directly updating the pretrained parameters, we propose to select weights important to downstream nlp tasks while discarding irrelevant ones. the selection mechanism consists of a set of binary masks, one learned per downstream task through end-to-end training. we show that masking, when being applied to pretrained language models like bert, roberta, and distilbert @xcite , achieves performance comparable to finetuning in tasks like part-of-speech tagging, named-entity recognition, sequence classification, and reading comprehension. this is surprising in that a simple subselection mechanism that does not change any weights is competitive with a training regime -finetuning -that can change the value of every single weight. we conduct detailed analyses revealing important factors and possible reasons for the desirable performance of masking. masking is parameter-efficient: only a set of 1bit binary masks needs to be saved per task after training, instead of all 32-bit float parameters in finetuning. this small memory footprint enables deploying pretrained language models for solving multiple tasks on edge devices. the compactness of masking also naturally allows parameter-efficient ensembles of pretrained language models. our contributions: (i) we introduce masking, a new scheme for utilizing pretrained language models by learning selective masks for pretrained weights, as an efficient alternative to finetuning. we show that masking is applicable to models like bert/roberta/distilbert, and produces performance on par with finetuning. (ii) we carry out extensive empirical analysis of masking, shedding light on factors critical for achieving good performance on eleven diverse nlp tasks. (iii) we study the binary masked language models' loss landscape and language representations, revealing potential reasons why masking has task performance comparable to finetuning. we propose to directly pick parameters appropriate to a downstream task, by learning selective binary masks via end-to-end training. keeping the pretrained parameters untouched, we solve several downstream nlp tasks with minimal overhead. binary networks and network pruning. binary masks can be trained using the "straightthrough estimator" @xcite . @xcite , @xcite , @xcite , inter alia, apply this technique to train efficient binarized neural networks. we use this estimator to train selective masks for pretrained language model parameters. investigating the lottery ticket hypothesis (frankle and carbin, 2018) of network pruning @xcite @xcite @xcite , @xcite find that applying binary masks to a neural network is a form of training the network. @xcite propose to search neural architectures for reinforcement learning and image classification tasks, without any explicit weight training. this work inspires our masking scheme (which can be interpreted as implicit neural architecture search @xcite ): applying the masks to a pretrained language model is similar to finetuning, yet is much more parameter-efficient. perhaps the closest work, mallya et al. (2018) apply binary masks to cnns and achieve good performance in computer vision. we learn selective binary masks for pretrained language models in nlp and shed light on factors important for obtaining good performance. @xcite explicitly update weights in a task-specific classifier layer. in contrast, we show that end-to-end learning of selective masks, consistently for both the pretrained language model and a randomly initialized classifier layer, achieves good performance. radiya-dixit and wang (2020) investigate finetuning of bert by employing a number of techniques, including what they call sparsification, a method similar to masking. their focus is analysis of finetuning bert whereas our goal is to provide an efficient alternative to finetuning. ## datasets and setup datasets. we present results for masking bert, roberta, and distilbert in part-of-speech tagging, named-entity recognition, sequence classification, and reading comprehension. we experiment with part-of-speech tagging (pos) on penn treebank @xcite , using @xcite 's train/dev/test split. for named-entity recognition (ner), we conduct experiments on the conll-2003 ner shared task @xcite . for sequence classification, the following glue tasks @xcite are evaluated: stanford sentiment treebank (sst2) @xcite , microsoft research paraphrase corpus (mrpc) @xcite , corpus of linguistic acceptability (cola) @xcite , recognizing textual entailment (rte) @xcite , and question natural language inference (qnli) @xcite . in addition, we experiment on sequence classification datasets that have publicly available test sets: the 6-class question classification dataset trec @xcite , the 4-class news classification dataset ag news (ag) @xcite , and the binary twitter sentiment classification task semeval-2016 4b (sem) @xcite . we experiment with reading comprehension on swag @xcite using the official data splits. we report matthew's correlation coefficient (mcc) for cola, micro-f1 for ner, and accuracy for the other tasks. setup. due to resource limitations and in the spirit of environmental responsibility @xcite , we conduct our experiments on the base models: bert-baseuncased, roberta-base, and distilbert-baseuncased. thus, the bert/roberta models we use have 12 transformer blocks (0-11 indexed) producing 768-dimension vectors; the distilbert model we use has the same dimension but contains 6 transformer blocks (0-5 indexed). we implement our models in pytorch @xcite with the huggingface framework @xcite . throughout all experiments, we limit the maximum length of a sentence (pair) to be 128 after wordpiece tokenization. following @xcite , we use the adam (kingma and ba, 2014) optimizer of which the learning rate is a hyperparameter while the other parameters remain default. we carefully tune the learning rate for each setup: the tuning procedure ensures that the best learning rate does not lie on the border of our search grid, otherwise we extend the grid accordingly. the initial grid is {1e-5, 3e-5, 5e-5, 7e-5, 9e-5}. for sequence classification and reading comprehension, we use [cls] as the representation of the sentence (pair). following devlin et al. (2019) , we formulate ner as a tagging task and use a linear output layer, instead of a conditional random field layer. for pos and ner experiments, the representation of a tokenized word is its last wordpiece @xcite . note that a 128 maximum length of a sentence for pos and ner means that some word-tag annotations need to be excluded. appendix §a shows our reproducibility checklist containing more implementation and preprocessing details. ## conclusion we have presented masking, an efficient alternative to finetuning for utilizing pretrained language models like bert/roberta/distilbert. instead of updating the pretrained parameters, we only train one set of binary masks per task to select critical parameters. extensive experiments show that masking yields performance comparable to finetuning on a series of nlp tasks. leaving the pretrained parameters unchanged, masking is much more memory efficient when several tasks need to be solved. intrinsic evaluations show that binary masked models extract valid and generalizable representations for downstream tasks. moreover, we demonstrate that the minima obtained by finetuning and masking can be easily connected by a line segment, confirming the effectiveness of applying masking to pretrained language models. our code is available at: https://github.com/ ptlmasking/maskbert . future work may explore the possibility of applying masking to the pretrained multilingual encoders like mbert @xcite and xlm @xcite . also, the binary masks learned by our method have low sparsity such that inference speed is not improved. developing methods improving both memory and inference efficiency without sacrificing task performance can open the possibility of widely deploying the powerful pretrained language models to more nlp applications.