W hen BERT P lays the L ottery, A ll T ickets A re W inning (2020)

Abstract: Large Transformer-based models were shown to be reducible to a smaller number of self-attention heads and layers. We consider this phenomenon from the perspective of the lottery ticket hypothesis, using both structured and magnitude pruning. For fine-tuned BERT, we show that (a) it is possible to find subnetworks achieving performance that is comparable with that of the full model, and (b) similarly-sized subnetworks sampled from the rest of the model perform worse. Strikingly, with structured pruning even the worst possible subnetworks remain highly trainable, indicating that most pre-trained BERT weights are potentially useful. We also study the “good” subnetworks to see if their success can be attributed to superior linguistic knowledge, but find them unstable, and not explained by meaningful self-attention patterns.

## introduction much of the recent progress in nlp is due to the transfer learning paradigm in which transformerbased models first try to learn task-independent linguistic knowledge from large corpora, and then get fine-tuned on small datasets for specific tasks. however, these models are overparametrized: we now know that most transformer heads and even layers can be pruned without significant loss in performance @xcite @xcite . one of the most famous transformer-based models is @xcite . it became a must-have baseline and inspired dozens of studies probing it for various kinds of linguistic information @xcite . we conduct a systematic case study of finetuning bert on glue tasks @xcite from the perspective of the lottery ticket hypothesis * equal contribution @xcite . we experiment with and compare magnitude-based weight pruning and importance-based pruning of bert self-attention heads @xcite , which we extend to multi-layer perceptrons (mlps) in bert. with both techniques, we find the "good" subnetworks that achieve 90% of full model performance, and perform considerably better than similarlysized subnetworks sampled from other parts of the model. however, in many cases even the "bad" subnetworks can be re-initialized to the pre-trained bert weights and fine-tuned separately to achieve strong performance. we also find that the "good" networks are unstable across random initializations at fine-tuning, and their self-attention heads do not necessarily encode meaningful linguistic patterns. ## related work multiple studies of bert concluded that it is considerably overparametrized. in particular, it is possible to ablate elements of its architecture without loss in performance or even with slight gains @xcite @xcite . this explains the success of multiple bert compression studies @xcite @xcite . while nlp focused on building larger transformers, the computer vision community was exploring the lottery ticket hypothesis (lth: @xcite @xcite . it is formulated as follows: "dense, randomly-initialized, feed-forward networks contain subnetworks (winning tickets) that -when trained in isolation -reach test accuracy comparable to the original network in a similar number of iterations" @xcite . the "winning tickets" generalize across vision datasets @xcite , and exist both in lstm and transformer models for nlp @xcite . task dataset train dev metric cola corpus of linguistic acceptability judgements (warstadt et al., 2019) 10k 1k matthews sst-2 the stanford sentiment treebank @xcite ) 67k 872 accuracy @xcite 4k n/a accuracy @xcite 7k 1.5k pearson qqp quora question pairs 1 @xcite 400k n/a accuracy mnli the multi-genre nli corpus (matched) @xcite 393k 20k accuracy qnli question nli @xcite 108k 11k accuracy rte recognizing textual entailment @xcite @xcite 2.7k n/a accuracy wnli winograd nli @xcite 706 n/a accuracy ideally, the pre-trained weights would provide transferable linguistic knowledge, fine-tuned only to learn a given task. but we do not know what knowledge actually gets used for inference, except that bert is as prone as other models to rely on dataset biases @xcite @xcite @xcite . at the same time, there is vast literature on probing bert architecture blocks for different linguistic properties @xcite . if there are "good" subnetworks, then studying their properties might explain how bert works. ## methodology all experiments in this study are done on the "bert-base lowercase" model from the transformers library @xcite . it is fine-tuned 2 on 9 glue tasks, and evaluated with the metrics shown in ## interpreting bert's subnetworks in subsection 4.2 we showed that the subnetworks found by m-and s-pruning behave similarly in finetuning. however, s-pruning has an advantage in that the functions of bert architecture blocks have been extensively studied (see detailed overview by @xcite ). if the better performance of the "good" subnetworks comes from linguistic knowledge, they could tell a lot about the reasoning bert actually performs at inference time. ## discussion this study confirms the main prediction of lth for pre-trained bert weights for both m-and s-pruning. an unexpected finding is that with spruning, the "random" subnetworks are still almost as good as the "good" ones, and even the "worst" ones perform on par with a strong baseline. this suggests that the weights that do not survive pruning are not just "inactive" @xcite . an obvious, but very difficult question that arises from this finding is whether the "bad" subnetworks do well because even they contain some linguistic knowledge, or just because glue tasks are overall easy and could be learned even by random bert (kovaleva et al., 2019), or even any sufficiently large model. given that we did not find even the "good" subnetworks to be stable, or preferentially containing the heads that could have interpretable linguistic functions, the latter seems more likely. furthermore, should we perhaps be asking the same question with respect to not only subnetworks, but also full models, such as bert itself and all the follow-up transformers? there is a trend to automatically credit any new state-of-the-art model with with better knowledge of language. however, what if that is not the case, and the success of pretraining is rather due to the flatter and wider optima in the optimization surface (hao et al., 2019)? can similar loss landscapes be obtained from other, non-linguistic pre-training tasks? there are initial results pointing in that direction: papadimitriou and jurafsky (2020) report that even training on midi music is helpful for transfer learning for lm task with lstms. ## conclusion this study systematically tested the lottery ticket hypothesis in bert fine-tuning with two pruning methods: magnitude-based weight pruning and importance-based pruning of bert self-attention heads and mlps. for both methods, we find that the pruned "good" subnetworks alone reach the performance comparable with the full model, while the "bad" ones do not. however, for structured pruning, even the "bad" subnetworks can be finetuned separately to reach fairly strong performance. the "good" subnetworks are not stable across finetuning runs, and their success is not attributable exclusively to non-trivial linguistic patterns in individual self-attention heads. this suggests that most of pre-trained bert is potentially useful in fine-tuning, and its success could have more to do with optimization surfaces rather than specific bits of linguistic knowledge.