Breaking through Deterministic Barriers: Randomized Pruning Mask Generation and Selection (2023)

Abstract: It is widely acknowledged that large and sparse models have higher accuracy than small and dense models under the same model size constraints. This motivates us to train a large model and then remove its redundant neurons or weights by pruning. Most existing works pruned the networks in a deterministic way, the performance of which solely depends on a single pruning criterion and thus lacks variety. Instead, in this paper, we propose a model pruning strategy that first generates several pruning masks in a designed random way. Subsequently, along with an effective mask-selection rule, the optimal mask is chosen from the pool of mask candidates. To further enhance efficiency, we introduce an early mask evaluation strategy, mitigating the overhead associated with training multiple masks. Our extensive experiments demonstrate that this approach achieves state-of-the-art performance across eight datasets from GLUE, particularly excelling at high levels of sparsity.

## introduction one of the main challenges in deploying large neural networks (such as bert @xcite and gpt-3 @xcite ) in production is the huge memory footprint and computational costs. meanwhile, studies show that large and sparse models often yield higher accuracy than small but dense models @xcite . as a result, pruning has been popularized to dramatically reduce memory size and computational power consumption with little to no performance degradation. @xcite @xcite @xcite @xcite . figure 1 : weight distribution in a feedforward layer of bert base at various sparsity levels (0.52 and 0.83), corresponding to pruning thresholds τ = 0.027 and τ = 0.055. notably, around 29% of the weights lie within the range [ 2 3 τ , 4 3 τ ]. this observation puts into question the efficacy of magnitude-based pruning, as these weights, despite their proximity to the threshold, might play a crucial role in maintaining the model's accuracy. this suggests that directly eliminating weights with smaller magnitudes could potentially lead to a suboptimal pruning strategy. 2020), namely to remove the elements with the smallest magnitude. here the magnitude refers to not only the weights but also the output sensitivity, gradients, or hessian matrices of the training loss @xcite @xcite @xcite @xcite . while magnitude-based pruning can generate state-of-theart results in a wide range of tasks, its pruning strategy is deterministic and solely depends on a single criterion, which lacks variety (we demonstrate this more thoroughly in the next paragraph). furthermore, magnitude-based pruning is proven not optimal at high-level sparsity @xcite . to further improve the pruning performance, @xcite ; @xcite ; @xcite ; @xcite ; @xcite try to enlarge the search space of sparse architecture with regularization-based methods, which are non-deterministic. they design fine-designed l 0 or l 1 penalty terms added to the loss function. in this way, the model proactively shrinks some of the weights until they don't contribute to the final loss. regularization-based methods can achieve noticeably better results than magnitude-based methods, especially at high-level sparsity @xcite . however, this line of work often suffers from a non-convex landscape and is challenging to optimize with extra hyperparameters. in parallel, @xcite ; @xcite adopt a more aggressive strategy to prune elements in a completely random way. their methods demonstrate the competitive effect in small datasets (such as cifar-10, cifar-100 @xcite ) but fail in large datasets (such as ima-genet @xcite ). different from these works, this paper introduces a mildly random pruning method that brings a controllable degree of randomness into the pruning mask generation procedure. we demonstrate the weakness of magnitudebased pruning in figure 1 . it presents the weight distribution of a feedforward layer of bert @xcite . define τ as the pruning boundary. we consider two scenarios: τ = 0.027 and τ = 0.055, leading to sparsity levels of 0.52 and 0.83, respectively. as shown in figure 1 , a large portion of weights (≈ 29%) falls into the range of [ 2 3 τ , 4 3 τ ], which cannot be overlooked because that it is unclear if the pruned weights close to the threshold contribute less than the kept weights in the final accuracy. the weights with smaller magnitudes can still be crucial, especially when dealing with edge cases or infrequent situations. proximity between weights can intensify decisionmaking challenges. this is why the direct removal of weights with smaller magnitudes is sub-optimal, as also demonstrated in @xcite . based on the above observations, we investigate the following questions in this paper: question 1. which is better for pruning? a deterministic way or a randomized way? previous literature has not reached a consistent conclusion. while @xcite and @xcite have provided evidence that random pruning can yield competitive or better results compared to deterministic methods, this finding does not consistently hold true for larger datasets. moreover, these results have not been universally extended to language models. we conjecture that their methods introduce unbridled randomness but do not provide any effective negative feedback. moreover, exploring the extent of introduced randomness in a principled way has also not been explored in the previous literature. in this paper, we study and extend the above question systematically. question 2. can we design a consistently effective randomized pruning method? this paper answers the above question with the following contribution. first, we propose a randomized pruning mask generation strategy that can introduce controllable randomness in a principled way. second, we design mask candidate selection strategy (mcss) to choose the optimal mask from the pool of mask candidates, ensuring the introduced randomness always guides pruning in a beneficial direction. third, to further enhance efficiency, we introduce early mask evaluation pipeline (emep) to mitigate the overhead associated with training under multiple pruning masks. last, we offer an empirical guidance for randomized pruning on bert base and bert large . our results show a consistent accuracy boost (0.1%∼2.6%) on the glue benchmark, outperforming other stateof-the-art pruning techniques at a 16x compression rate. notably, our approach showcases even more significant enhancements (2%∼4%) at extreme sparsity levels like the 100x compression rate. ## methodology in this section, we first rethink the traditional deterministic pruning method and introduce our basic idea and method. following that, we elaborate on the details of our randomized pruning mask generation and selection strategy. the architecture of our strategy is depicted in figure 2 , and the detailed procedure is outlined step-by-step in algorithm 1. ## experiments we evaluate the effectiveness of our pruning strategy in a wide range of natural language understanding tasks. following previous work, we use bert as our backbone and then apply different pruning methods to compare their performance. ## related work a number of researchers have explored pruning in bert. @xcite prunes the model with @xcite 's first-order importance metric and proves that unstructured magnitudebased pruning always produces sparser and higheraccuracy models than structured pruning. @xcite and @xcite both argue that pruning in the pre-trained stage is better than finetuning stage because there is no need to prune for each downstream task. they also conclude that knowledge in sparse training can be transferred as well as dense models. in contrast, @xcite find that pruning at the pre-trained stage has huge computational problems while pruning at the finetuning stage can save computational efforts and keep accuracy simultaneously. these pruning methods are based on the magnitude and prune weights in a deterministic way. in parallel, a linear of works defeats magnitude-based methods at highlevel sparsity by applying a non-deterministic way: regularization-based pruning. specifically, a finedesigned l 0 or l 1 penalty terms are added to the loss function. then the model proactively shrinks some of the weights until they do not contribute to the final loss. the regularization-based method can generally achieve significantly better results than the magnitude-based methods, especially at highlevel sparsity @xcite . however, penalty terms can introduce additional local minima to the loss function and are difficult to navigate optimization. on the other hand, there is a lack of research on random pruning applied to transformer-based models (such as bert) in previous studies. therefore, our paper complements the gap in this area. ## conclusion this paper presents a non-deterministic model pruning method, introducing controllable randomness by generating binary masks in a specific random fashion. coupled with our specific mask candidate selection rule, our method exhibits significant effectiveness in enhancing model accuracy in pruning, particularly at high levels of sparsity.