Combining Compressions for Multiplicative Size Scaling on Natural Language Tasks (2022)

Abstract: Quantization, knowledge distillation, and magnitude pruning are among the most popular methods for neural network compression in NLP. Independently, these methods reduce model size and can accelerate inference, but their relative benefit and combinatorial interactions have not been rigorously studied. For each of the eight possible subsets of these techniques, we compare accuracy vs. model size tradeoffs across six BERT architecture sizes and eight GLUE tasks. We find that quantization and distillation consistently provide greater benefit than pruning. Surprisingly, except for the pair of pruning and quantization, using multiple methods together rarely yields diminishing returns. Instead, we observe complementary and super-multiplicative reductions to model size. Our work quantitatively demonstrates that combining compression methods can synergistically reduce model size, and that practitioners should prioritize (1) quantization, (2) knowledge distillation, and (3) pruning to maximize accuracy vs. model size tradeoffs.

## introduction as increasingly large models dominate natural language processing (nlp) benchmarks, model compression techniques have grown in popularity @xcite @xcite . for example, quantization @xcite @xcite lowers bit precision of network weights to reduce memory usage and accelerate inference @xcite . knowledge distillation (kd; @xcite ), which trains a student neural network using the logits (or representations) of a teacher network, is used widely to transfer knowledge to smaller models @xcite @xcite . pruning identifies weights which can be omitted at test time without significantly degrading performance. some pruning methods remove individual weights according to magnitudes or other heuristics @xcite @xcite , while others remove structured blocks of weights or entire attention heads @xcite @xcite . recent work has begun combining these compression methods for improved results. @xcite , @xcite , and @xcite have used knowledge distillation with pruning or low-bit quantization to fine-tune bert. as practitioners look to combine methods more generally, new research is needed to compare their empirical value and study interactions. this work addresses the questions: (1) which popular compression methods or combinations of methods are usually most effective? (2) when combining methods, are their benefits complementary or diminishing? we address these questions by computing accuracy vs. model size tradeoff curves for six pretrained bert sizes fine-tuned on eight glue tasks @xcite , applying each of eight possible subsets of quantization-aware-training (qat), knowledge distillation (kd), and magnitude pruning (mp). our main findings are as follows: ## methods in our work, we study three common model compressions: quantization-aware-training, knowledge distillation, and magnitude pruning. we prioritize performant, broadly applicable approaches with accessible implementations, so our findings are most useful to practictioners. hyperparameters and additional method details are in appendices a & c. ## results experiments for six bert architecture sizes and eight glue tasks foot_0 , we tested every possible subset of compression methods: no compression (baseline), qat, kd, mp, qat+kd, qat+mp, kd+mp, and qat+kd+mp. for each of 576 experiment settings, we log the max glue development set accuracy across twelve hyperparameter configurations and five repetitions of each configuration. in figure 1 , we plot mean glue accuracy across all eight tasks on the y-axis against decreasing model size on the x-axis, for each compression combination. the curves without pruning include six points, one for each architecture size from large to tiny. with pruning, there are twice as many points, as each architecture is pruned to either 40% or 60% encoder sparsity. results split by task are available in appendix figure a1 . individually, qat and kd are most effective. for all architectures, qat (blue) reduces model size by 4× while minimally reducing accuracy: the largest drop is -0.6% for bert-base (supporting zafrir et al. ( 2019 )), while other architecture sizes are nearly unaffected. kd (orange) does not reduce a given architecture's size, but instead yields a consistent boost to accuracy, especially for smaller architectures. this upward shift on the accuracy-model size curve means that larger models can be downsized more effectively: e.g., large's baseline accuracy is matched by the kd version of base, and kd small outperforms baseline medium. compared to qat or kd, mp (green) is only modestly helpful. typically, 40% of encoder weights can be removed without much impact, but pruning 60% (i.e., the second point in each set of two) degrades accuracy. also, removing 40% of encoder weights corresponds to @xmath0. therefore, while mp can be helpful for large and base, it cannot significantly compress small architectures, which have a higher percentage of their weights in embeddings. used together, kd mitigates accuracy losses from both qat and mp. moving to combinations of compression methods, we find that the most successful pair combines qat and kd (red), which yields qat's 4× memory reduction while retaining the improved accuracy over baseline from kd. meanwhile, qat+mp (purple) does poorly: though large and base can prune 40% of weights while retaining accuracy, when they are pruned and quantized, they have lower accuracy than when they are only quantized. this result suggests that pruning specifically damages accuracy with quantization: practitioners should expect additive (or worse) accuracy degradation when combining qat and mp. on the other hand, with kd+mp (brown), 40% weights can be pruned while retaining the accuracy boost from kd, for all architectures. thus, kd mitigates accuracy losses from both mp and qat. this result still holds when we combine all three methods (black). with qat+kd+mp, 40% of encoder weights for large and 60% for base (and smaller) can be removed while matching the accuracy of qat+kd. in our experiments, kd completely mitigates the compounding losses from qat+mp and even improves accuracy. kd enables deeper compression when practitioners combine methods. combining methods yields super-multiplicative compression ratios. building on our qualitative findings, we were interested in quantitative estimates for how much each method allows us to compress each architecture size. so, we compute compression ratios, i.e., the maximal size reduction factor possible while preserving accuracy to within 0.5% of baseline base with qat, kd, and 40% mp (76 mb) is the smallest model within 0.5% of that, at 92.6% accuracy. therefore, bert-large's compression ratio on qnli is 1341 76 = 17.6×, and averaging this value across tasks yields a net ratio of 18.1× (topright, as before, qat usually retains accuracy and yields a 4× size reduction. however, because there are a few tasks for which qat causes @xmath1.5% accuracy drop 2 , the task-averaged compression ratios for large and base of 3.6× and 2.6×. kd also has high mean compression ratios, because it often boosts small architectures' accuracies to match larger baseline architectures. mp yields 1.7× compression (from 40% pruning) for large and base, and even less for the smaller architectures. when combined, we observe synergistic compression between qat and kd. we might expect strong diminishing returns from combining methods, but even in their absence, we would expect independent compression ratios to multiply. strikingly, though, we often see super-multiplicative model size reductions with qat+kd: e.g. base, 5.9 > 2.6 • 1.8 = 4.7; medium: 8.0 > 2.9 • 2.0 = 5.8. for kd+mp, the ratios multiply for large and base. also, while pruning was originally ineffective for medium and smaller, kd+mp appears to make pruning more effective, again with super-multiplicative compression (e.g. for medium: 3.5 > 2.0 • 1.2 = 2.4). these findings show that kd mitigates the drop in accuracy from quantization @xcite and pruning @xcite , supporting qualitative findings from prior literature. this mitigation effect, combined with the accuracy boost that kd provides, yields joint compressions that are more effective than the sum of their parts. remarkably, with all three compressions, we still see 2 those tasks would be considered to have 1× compression. ## discussion and future work we examine the relative benefits and interactions of three widely-used compression methods in nlp: knowledge distillation, int8 quantization-awaretraining, and magnitude pruning. though multiple techniques are increasingly used simultaneously, little work has studied the empirical interactions between them. across six architecture sizes and eight glue tasks, we find that int8 quantization is most beneficial, and combining quantization with kd mitigates any of its occasional accuracy drops. quantization and pruning yield diminishing returns, with the two methods exacerbating accuracy losses when used together. however, when all three methods are combined, kd restores accuracy above baseline, yielding 18× and 11× net compression for bert-large and base. we quantitatively confirm that kd's model size improvements are complementary (often super-multiplicative) with quantization and pruning. given these benefits, using larger, more recent glue winners as kd teachers may yield further gains (e.g. ernie, @xcite ). we also hope that our observed benefits inspire authors of new compression techniques to evaluate complementarity with existing methods. in future work, we hope to measure accuracy vs. inference time tradeoffs, and compare these results to our findings on model size. though there is work on accelerated inference with int8 quantization @xcite or pruning @xcite , it is not clear how these speedups stack. profiling compression combinations on specialized hardware would be an informative avenue to explore next.