Structured Pruning for Efficient Generative Pre-trained Language Models (2023)

Abstract: The increasing sizes of large generative Pre-trained Language Models (PLMs) hinder their deploymentin real-world applications. To obtain efficient PLMs, previous studies mostly focus on pruning the attention heads and feed-forward networks (FFNs) of the Transformer. Nevertheless, we find that in generative PLMs, the hidden dimension shared by many other modules (e.g., embedding layer and layer normalization) contains persistent outliers regardless of the network input. This study comprehensively investigates the structured pruning of generative PLMs with all the above compressible components. To identify redundant network structures, we assign learnable masks over compressible components followed by sparse training. Various sizes of PLMs can be flexibly extracted via different thresholds, and are then task-specifically fine-tuned for further improvement. Extensive experiments on language modeling, summarization and machine translation validate the effectiveness of the proposed method. For example, the pruned BART brings 1.51x/6.96x inference speedup on GPU/CPU with 67% size reduction, and can be further combined with quantization for more than 25 \times compression.

## introduction large-scale generative pretrained language models (plms) @xcite @xcite show remarkable performance on various tasks. however, their increasing sizes also lead to expensive memory and computation, hindering their deployment in real applications. recent attempts @xcite @xcite @xcite propose to compress generative plms models by quantization. however, hardware-dependent low-bit kernels need to be specially developed for real inference speedup. com-pared to quantization, structured pruning methods prune parts of the model structures without requiring designing extra operators to achieve inference speedup and run-time memory saving. recently, @xcite show that the feed-forward networks (ffns) of gpt models can be pruned to smaller widths, and @xcite compress the bart models by combining layer pruning and model quantization for a higher compression rate. however, these models consider only limited components for pruning, i.e., the ffns or transformer layers, which can be restrictive for various deployment requirements. in this work, we propose a new structured pruning framework named simple (sparsity-induced mask learning for pruning generative pre-trained language models), which offers a wider range of compressible components. aside from the attention heads and the width of ffns commonly used for structured pruning of discriminative plms, we also propose to prune the hidden dimension, to further push the trade-off between performance and model size. it is motivated by the observation that persistent outliers exist in the hidden dimension of both decoder-only and encoder-decoder generative plms. the observation implies that the hidden dimension may be slimmed sharply with a slight performance drop. additionally, as the dimension of the hidden state is shared by many modules, e.g., embedding layer, attention heads, ffn and layer normalization, the model size thus can be collectively slimmed for further compression. the crux of pruning lies in ranking the importance of different compressible components. towards that end, we assign learnable masks over the output of all compressible components. these masks are optimized in an end-to-end fashion together with a sparsity-induced penalty. unlike conventional pruning criteria based on magnitudes @xcite or gradients @xcite , these masks can be mathematically interpreted to prune away components shrinking towards zero during the training. moreover, the learning of masks is one-shot, i.e., one can flexibly obtain various pruned models by different thresholds over the learned mask values. to mitigate the accumulated compression error, a causal distillation objective is proposed to fine-tune the pruned networks for further improvement on downstream tasks. we verify the efficacy of simple on various generation tasks (i.e., language modeling, summarization and machine translation) and network architectures (i.e., gpt2 and bart/mbart). empirical results show that the proposed simple outperforms other pruning methods, significantly speeds up the inference, and can be combined with quantization for more aggressive compression. ## methodology in this section, we elaborate on the proposed sim-ple for structurally pruning generative plms. in section 3.1, we include the hidden dimension as compressible components in addition to the attention heads and ffn neurons, due to the existence of persistent outliers. then, we introduce sparse learning of masks together with their mathematical interpretations in section 3.2. the structurally pruned models are followed by task-specific fine-tuning, as detailed in section 3.3. an overall framework of simple is depicted in figure 1 . ## conclusions we propose simple, an efficient method to structurally prune generative pre-trained language models. simple includes attention heads, the neurons in the intermediate layer of the feed-forward network, and the hidden dimension as compressible components. by learning the masks of these compressible components through a sparsity-induced objective, different sized pruned models can be obtained, and further fine-tuned with a causal distillation objective for better performance. empirical evaluations on different generative tasks, model architectures, and pruning configurations demonstrate the efficacy of the proposed method.