Much of the recent success in NLP is due to the large Transformer-based models such as BERT (Devlin et al, 2019). However, these models have been shown to be reducible to a smaller number of self-attention heads and layers. We consider this phenomenon from the perspective of the lottery ticket hypothesis  . For fine-tuned BERT, we show that (a) it is possible to find a subnetwork of elements that achieves performance comparable with that of the full model, and (b) similarly-sized subnetworks sampled from the rest of the model perform worse. However, the "bad" subnetworks can be fine-tuned separately to achieve only slightly worse performance than the "good" ones , indicating that most weights in the pre-trained BERT  are potentially useful. We also show that the "good" subnetworks vary considerably across GLUE tasks, opening up the possibilities to learn what knowledgeBERT actually uses at inference time .
Large Transformer-based models were shown to be reducible to a smaller number of self-attention heads and layers. We consider this phenomenon from the perspective of the lottery ticket hypothesis , using both structured and magnitude pruning . For fine-tuned BERT, we show that (a) it is possible to find subnetworks achieving performance that is comparable with that of the full model, and (b) similarly-sized subnetworks sampled from the rest of the model perform worse. Strikingly, with structured pruning even the worst possible subnetworks remain highly trainable , indicating that most pre-trained BERT weights are potentially useful. We also study the "good" subnetworks to see if their success can be attributed to superior linguistic knowledge, but find them unstable, and not explained by meaningful self-attention patterns .
R
Much of the recent success in NLP is due to the large
Large
R
such as BERT (Devlin et al, 2019). However, these models have been
were
A
, using both structured and magnitude pruning
R
a subnetwork of elements that achieves performance
subnetworks achieving performance that is
R
However, the "bad" subnetworks can be fine-tuned separately to achieve only slightly worse performance than the "good" ones
Strikingly, with structured pruning even the worst possible subnetworks remain highly trainable
D
weights in the
A
weights
R
show that
study
R
vary considerably across GLUE tasks, opening up the possibilities to learn what knowledgeBERT actually uses at inference time
to see if their success can be attributed to superior linguistic knowledge, but find them unstable, and not explained by meaningful self-attention patterns