Finding the Dominant Winning Ticket in Pre-Trained Language Models (2022)

Abstract: The Lottery Ticket Hypothesis suggests that for any over-parameterized model, a small subnetwork exists to achieve competitive performance compared to the backbone architecture. In this paper, we study whether there is a winning lottery ticket for pre-trained language models, which allow the practitioners to fine-tune the parameters in the ticket but achieve good downstream performance. To achieve this, we regularize the fine-tuning process with L1 distance and explore the subnetwork structure (what we refer to as the “dominant winning ticket”). Empirically, we show that (a) the dominant winning ticket can achieve performance that is comparable with that of the full-parameter model, (b) the dominant winning ticket is transferable across different tasks, (c) and the dominant winning ticket has a natural structure within each parameter matrix. Strikingly, we find that a dominant winning ticket that takes up 0.05% of the parameters can already achieve satisfactory performance, indicating that the PLM is significantly reducible during fine-tuning.

## introduction pre-trained language models (plms) have shown significant performance on various natural language processing (nlp) tasks @xcite . however, as the number of model parameters gets huge, fine-tuning such models becomes inefficient. many previous works target parameter-efficient fine-tuning approaches by freezing the plm parameters. one can either freeze a subset of the parameters @xcite and *corresponding authors: dongyan zhao (zhaody@pku.edu.cn) and rui yan (ruiyan@ruc.edu.cn) figure 1 : an illustration of the structure of the dominant winning ticket. in parallel to this line of study, an emerging subfield has explored the possibility of training smaller subnetworks in place of the full models without hurting performance @xcite . among them, the lottery ticket hypothesis (lth) @xcite has attracted much attention. lth demonstrates that an overparameterized network contains "winning tickets" (small-scale subnetworks) that can 1) match the performance of the full model; and 2) outperform randomly sampled subnetworks of the same size. winning tickets have been verified to exist in plms @xcite . in this paper, we provide an interesting result by showing that a subnetwork, which we refer to as the dominant winning ticket, exists in the plm. it can make us to freeze all other parameters but only train the parameters of the subnetwork and obtain competitive performance for any downstream tasks. to achieve this, we fine-tune the plm on one task (e.g., mnli) and enforce the parameter weights close to their initial weights (pre-trained weights) by using the l1-distance penalty. this allows us to identify which parts of the model parameters change greater from the pre-trained weights during fine-tuning. we observe this ticket has some novel properties: we organize the paper as follows. after a brief overview of related works in section 2, we introduce how we are aware of the existence of the dominant winning ticket in plms and how we identify and extract it in section 3. then we experiment on the dominant winning ticket in section 4. in section 5, we do further discussions about the dominant winning ticket. section 6 is about some implications and future directions of the work. ## the dominant winning ticket in plms we start with a question: which parts of the model parameters are more important when adapting plms to downstream tasks? to answer this question, we design the l1-regularized fine-tuning approach and then analyze the l1-regularized weights and reveal the existence of the dominant winning ticket. figure 2 : visualization of out weighted parameters in roberta-large. each row is a model fine-tuned on different downstream tasks. each column is the parameter matrix of different layers. for example, the top left block represents the value projection matrix of the 5th transformer layer from the model fine-tuned with the mnli dataset. ("k", "q", "v", "o" stand for the key, query, value, and output projection metrics of the self-attention module, "fc1" and "fc2" stand for the successive two fully-connected transformation metrics in each layer.) ## testing the dominant winning ticket for fine-tuning the previous section is about how we identify and extract the dominant winning ticket. in this section, we discuss the properties of the dominant winning ticket by conducting systematic experiments. to extract the dominant winning tickets, we perform l1-regularized fine-tuning on mnli, qnli, sst-2, and qqp respectively. we find that the dominant winning tickets corresponding to different downstream tasks look very close to each other (detailed statistics can be found in appendix a.1), which matches the observation in section 3.2. so, for simplicity, we regard the subnetwork extracted from mnli as the standard dominant winning ticket. our evaluation experiments are conducted upon it. we compare the dominant winning ticket with the subnetwork that has the same size as the dominant winning ticket, denoted as dominantk and random-k respectively. the dimensions of random-k are chosen from uniform distribution. as the scale of the extracted subnetwork is adjustable by choosing different k, we consider two different compression ratios, i.e., @xmath0. ## discussions the lottery ticket hypothesis (lth) states that dense, randomly-initialized networks contain subnetworks (winning tickets) that-when trained in isolation-reach performance comparable to the original network. initialization is an important factor in lth as the winning ticket extracted from a specific initialization generally behaviors poorly on other random reinitialized networks @xcite . this phenomenon is often ignored when discussing lth in plms as the initialization of plms is deterministic. plms initialize via pre-training weights, which contain rich information and are potentially useful. as a result, many subnetworks inside plms are potential winning tickets indeed, @xcite claimed that even the "bad" subnetworks in bert ("good" and "bad") have acceptable performance. we further argue that any randomly sampled subnetwork with the sparsity of 95% can serve as the winning ticket. we randomly select 5% of the parameters in each matrix and freeze the rest of the model, surprisingly finding that the performance is comparable with full-parameter fine-tuning. as seen in however, we still argue that not all tickets are winning. the smaller the ticket is, the rarer the winning ticket is. when the sparsity of the subnetwork increases, the performance of subnetworks begins to vary. for roberta-large, at the sparsity of 99.95%, most of the "ticket" fail to win while a well-selected subnetwork (the dominant winning ticket) can suffer little performance drops. in this sense, the dominant winning ticket we found can be regarded as the smallest winning ticket. ## implications and future work the dominant winning ticket can be utilized for parameter-efficient fine-tuning. as the dominant winning ticket is stable across different tasks, when deploying plms on different scenarios, we only need to record the same small group of parameters. besides, benefiting from the structure of the ticket, we only need to store the optimizer states for certain dimensions of each parameter matrix. with decent code implementation, we can promisingly reduce memory usage and speed up the training process. in future work, we would examine the memory reduction and speedup abilities of the dominant winning ticket. besides the practical value, the dominant winning ticket raises some interesting questions about plms. now that the dominant winning ticket is intrinsically determined by the pre-trained weights, then how does the subnetwork emerge during pretraining? another worth investigating point lies in the natural structure of the ticket. why certain dimensions of parameter matrices behave so differently from others? we still know little about the mechanism behind these phenomenons. we aim to study these questions in future work. ## conclusion in this paper, we reveal the existence of the dominant winning ticket inside pre-trained models and introduce the l1-regularized fine-tuning to extract it. the dominant ticket is an extremely sparse subnetwork that can reach comparable performance with fine-tuning the whole model. we observe that the ticket has some novel properties. first, it is stable across different random seeds and tasks, which means once identified on one task, it can be transferred to other tasks with no performance loss. second, the ticket has a natural structure within each parameter matrix, and this structure is shared across layers. our study not only has practical values for parameter-efficient fine-tuning but also raises some questions about the pre-trained models.