What’s Hidden in a One-layer Randomly Weighted Transformer? (2021)

Abstract: We demonstrate that, hidden within one-layer randomly weighted neural networks, there exist subnetworks that can achieve impressive performance, without ever modifying the weight initializations, on machine translation tasks. To find subnetworks for one-layer randomly weighted neural networks, we apply different binary masks to the same weight matrix to generate different layers. Hidden within a one-layer randomly weighted Transformer, we find that subnetworks that can achieve 29.45/17.29 BLEU on IWSLT14/WMT14. Using a fixed pre-trained embedding layer, the previously found subnetworks are smaller than, but can match 98%/92% (34.14/25.24 BLEU) of the performance of, a trained Transformer _\text{small/base} on IWSLT14/WMT14. Furthermore, we demonstrate the effectiveness of larger and deeper transformers in this setting, as well as the impact of different initialization methods.

## introduction modern deep learning often trains millions or even billions of parameters @xcite @xcite to deliver good performance for a model. recently, @xcite ; @xcite demonstrated that these over-parameterized networks contain sparse subnetworks, when trained in isolation, that can achieve similar or better performance than the original model. furthermore, recent studies revisit the initialization stage of finding these subnetworks in vision models @xcite . such a mask, which is used to mask out a part of the entire network to those subnetworks, 1 we released the source code at https://github. com/sincerass/one_layer_lottery_ticket . in this work, we first validate that there exist subnetworks of standard randomly weighted transformers (reservoir transformers in @xcite ) that can perform competitively with fully-weighted alternatives on machine translation and natural language understanding tasks. with 50% randomized weights remaining, we found a subnetwork that can reach 29.45/17.29 bleu on iwslt14/wmt14, respectively. we also investigate the special case of finding subnetworks in one-layer randomly weighted transformers (see fig. 1 ). to obtain the subnetworks, we repeatedly apply the same randomized transformer layer several times with different supermasks. the resulting subnetwork of a one-layer randomly-weighted transformer has similar performance as the multi-layer counterparts with a 30% lower memory footprint. we also study the impact of different depths/widths of transformers along with the effectiveness of two initialization methods. finally, using the pre-trained embedding layers, we find that the subnetworks hidden in one layer randomly weighted transformer wide/wider are smaller than, but can match 98%/92% of the performance of, a trained transformer small/base on iwslt14/wmt14. we hope our findings can offer new insights for understanding transformers. ## methodology finding a supermask for randomly weighted transformer. in a general pruning framework, denote weight matrix as @xmath0, input as @xmath1; w). a subnetwork defined is f (x; w m), where @xmath2. to find the subnetwork for a randomly weighted network, @xmath3. following ramanujan et al. (2020), denote @xmath4, which is learnable during training. we keep top-k percents of weights by the importance score of s to compute m, i.e., note that top k is an undifferentiated function. to enable training of s, we use the straight-through gradient estimator @xcite , in which top k is treated as the identity in backpropagation. during inference, we can simply construct and store the binary supermask m and the floatingpoint w while dropping s for future usage. one-layer randomly weighted transformer. we use the transformer architecture (see @xcite for more details). for a general randomly weighted transformer model with supermask, there exist m l s and w l s for all layers @xmath5, ...l}. due to the natural property of layer stacking in transformers, all w l s have the same shape with the same initialization method. this leads to an unexplored question: "what's hidden in a one-layer (instead of l-layer) randomly weighted transformer?" let us use a toy example to explain why there is no need for l redundant w l s. assume that, for a random weighted matrix w l , the probability that it has a "good" subnetwork is p foot_0 . furthermore, assume that for two different layers, the probability that both have the "good" subnetworks is independent. then for l different layers, the probability that all w l s have the "good" subnetworks is p l . meanwhile, since w 1 has the same initialization method as w l , the probability that w 1 has a "good" subnetwork for l-th layer is also p. thus, for l different layers, the probability that using w 1 to generate all "good" subnetworks is also p l . in this paper, we investigate the scenario where one randomized layer is applied for l times repeatedly with l different supermasks. as a result, this can reduce the memory footprint since all supermasks can be stored in the binary format. for model architectures, we experiment with transformer small and transformer base , following the same setting as in @xcite : 6 encoder layers and 6 decoder layers on iwslt14 and wmt14. we also vary the depth and width of the transformer model on machine translation tasks. on iwslt14, we use 3 different random seeds and plot the mean @xmath6. all the embedding layers (including the final output projection layer) are also randomized and pruned unless otherwise specified. moreover, on all figures, the "fully-weighted model" denotes the standard full model (all weights remaining). machine translation results. in fig. 2 , we present results for directly pruning a randomly weighted transformer on iwslt14 and wmt14 tasks. specifically, we vary the ratio of remaining parameters in the randomized model. as can be seen, there is no significant performance difference between a one-layer random transformer versus a 6-layer standard random transformer across different percents of remaining weights on iwslt14 and wmt14. we also observe that having the remaining randomized weight percents approach 0 or 100 leads to the worst performance across the settings. this is expected since the outputs will be random when we have 100% randomized weights, and the model will not perform well when only limited weights are unpruned (close to 0%). the best performing subnetwork of a one-layer randomized transformer has 50% weights remained. connected to the search space of the employed method where we are choosing σ% out of 100% randomized weights, σ = 50 leads to the largest search space. effectiveness of pre-trained embeddding layers. embedding layers are critical since they can be viewed as the inputs for an nlp model, particularly, 3 https://github.com/pytorch/fairseq/ 4 we use the checkpoint from fairseq for transbase/big on wmt14, and transsmall on iwslt14 to obtain the pre-trained embedding layer for one-layer transbase/wider and one-layer transsmall. for one-layer transwide on iwslt14, we pre-train fully-weighted model and then dump the embedding layer. transdeep/deeper share the same embedding of the transsmall/base. 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 remaining parameter ratio 0 5 10 15 20 25 30 35 bleu score prune randomly weighted transformer on iwslt kaiming_uniform 1-layer random transformersmall xavier_uniform 1-layer random transformersmall 6-layer fully-weighted transformersmall figure 5: the effectiveness of different initialization. the deeper transformer can already achieve 79%/90% of the fully-weighted baseline models on wmt14/iwslt14, respectively. for wider models, those numbers even increase to 92%/98%. this is mainly due to the larger search space introduced by the larger weight matrix. another important point is that even when we increase/enlarge the depth/width of the model, the total memory consumption of these models is actually smaller than the standard baseline, since we only have one repeated layer and all the masks can be stored in a 1-bit setting. furthermore, we explore the effect of the different ratios of remaining parameters for different models on iwslt14 in fig. 4 . as can be seen, for the wider model, its performance is always better than the standard one across all different settings. however, for the deeper model, there is a sharp transition that happens at 50%-60% remaining parameters. the reason is that, given that our deeper model is twice as deep as the original, when we retain more random parameters (>50%), the probability that the layer has a good "subnetwork" decreases significantly. this will lead the final probability to be p 2l smaller (p @xmath7, which is much smaller than p l (see section 3). different initialization. weight initialization is one of the critical components to the success of 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 remaining parameter ratio 60 65 70 75 80 accuracy prune randomly weighted transformer on mnli 12-layer random robertabase 1-layer random robertabase 1-layer random robertalarge 12-layer fully-weighted robertabase figure 7: prune randomly weighted transformer performance on mnli. the random feature @xcite @xcite . we experiment with kaiming uniform @xcite and xavier uniform @xcite initialization methods, and we scale the standard deviation by 1/σ when we retain σ randomized weights. as shown in fig. 5 , the performance of the one-layer randomized transformer decreases when we switch to the xavier uniform. the degradation becomes larger when more randomized weights retain in the network. qqp and mnli results. on qqp and mnli, we experiment with roberta small and roberta large , following @xcite . we use the pre-trained embedding layer of roberta base/large @xcite . in fig. 6 and 7, we show consistent results on qqp and mnli, except that the best performing onelayer randomly weighted roberta is achieved when we retain 70% randomized weights, it reaches 79%/91% fully-weighted roberta base accuracy on qqp and mnli, respectively. the performance approaches 84%/92% of the afore-mentioned fully-weighted model performance when using the larger hidden size with one-layer randomly weighted roberta large . implementation details. we evaluate on iwslt14 de-en @xcite and wmt14 en-de @xcite for machine translation; qqp @xcite and multinlimatched (mnli) @xcite for natural language understanding. 5 we use 8 volta v100 gpus for wmt, and one v100 for iwslt, qqp, and mnli. the hyperparameters on iwslt14 and wmt14 for training a one-layer randomized transformer were set the same to the best-performing values from @xcite for training fully-weighted transformer. the qqp and mnli experiments followed @xcite . ## conclusions in this paper, we validate the existence of effective subnetworks in a one-layer randomly weighted transformer on translation tasks. hidden within a one-layer randomly weighted transformer wide/wider with fixed pre-trained embedding layers, we find there exist subnetworks that are smaller than, but can competitively match, the performance of a trained transformer small/base on iwslt14/wmt14.